Interconnection system

ABSTRACT

An interconnection system, apparatus and method is described for arranging elements in a network, which may be a data memory system, computing system or communications system where the data paths are arranged and operated so as to control the power consumption and data skew properties of the system. A configurable switching element may be used to form the interconnections at nodes, where a control signal and other information is used to manage the power status of other aspects of the configurable switching element. Time delay skew of data being transmitted between nodes of the network may be altered by exchanging the logical and physical line assignments of the data at one or more nodes of the network. A method of laying out an interconnecting motherboard is disclosed which reduces the complexity of the trace routing.

This application claims the benefit of U.S. Provisional Application Ser.No.:60/674,189, filed on Apr. 21, 2005, and U.S. Provisional ApplicationSer. No.:60/698,626, filed on Jul. 11, 2005, each application beingincorporated herein in its entirety by reference.

BACKGROUND

Generally, the evolution of computer and communications products hasbeen governed by such principles as “Moore's Law”, where both thedensity of the components increases, and the cost of such a componentdecreases with time. This has also often been accompanied by increasesin transfer and computation speed. These trends have often increased thepower dissipation density and required new solutions to either dissipatethe heat or operate in the high temperature environment. In high speedcircuits, both signal propagation delay and the design and manufacturingtolerances may place limitations on the realizable speeds attained inpractice.

Contemporary memory system architectures may demonstrate tradeoffsbetween cost, performance and the ability to upgrade, for example; thetotal memory capacity of the system. Memory capacity is commonlyupgraded via memory modules or cards having a connector/socketinterface. Often these memory modules are connected to a bus orinterconnecting wiring disposed on a backplane to utilize systemresources efficiently. In addition to upgradeability, many of thesecontemporary memory systems also require high throughput for bandwidthintensive applications, such as graphics.

System resources which may be considered in design include, for example,integrated circuit die area, package pins, signal line traces,connectors, backplane board area and power and response speed.

One popular type of memory module is a Dual In-line Memory Module(DIMM). The DIMM is a rectangular low-profile circuit board that haselectrical contact points arranged on both sides along one long edge.The contact points form electrical connections to the main board'smemory bus when the DIMM is inserted into a DIMM memory socket.

Memory systems provide an upgrade path through the usage of modules. Asocket and connector interface may be employed which allows each moduleto be removed and replaced by a memory module that is faster or includesa higher capacity, or merely for the replacement of a failed memorymodule. A memory system may be configured with unpopulated sockets orless than a full capacity of modules (i.e., empty sockets/connectors)and provided for increased capacity at a later time with memoryexpansion modules.

Memories used in computing and communications systems include, but arenot limited to, random access memory (RAM) of all types (e.g., S-RAM,D-RAM); programmable read only memory (PROM); electronically alterableread only memory (EPROM); flash memory, magnetic memories of all typesincluding Magnetoresistive Random Access Memory (MRAM), FerroelectricRAM (FRAM or FeRAM) as well as NRAM (Nanotube-based/Nonvolatile RAM) andPhase-change memory (PRAM), and magnetic disk storage media. Othermemories which may become suitable for use in the future include quantumdevices and the like.

There is a need for memory system and computing system architectures orinterconnect topologies that provide flexible and cost effective upgradecapabilities while providing high bandwidth to keep pace withmicroprocessor operating frequencies. Alternatively, lower powerconsumption, lower memory latency, and similar attributes for the sameor similar operational characteristics are also needed.

At present, the size and performance of memory systems may be limited bypower consumption, by cooling capacity associated with the powerconsumption and device density, by response time due to, for example,data skew, by the complexity of the circuit boards needed tointerconnect the memory elements, and by the economic costs of theseconsiderations. Improvements in memory and communications systemsarchitectures and apparatus are needed to mitigate some or all of theseproblems.

SUMMARY

An interconnect system for use with components of a computing ortelecommunications system is disclosed, the interconnect system having aplurality of nodes, and each node has a port having at least one of aninput link or an output link. Each link has lanes (“lines”). A firstlane of a first node transmits control data, and a second node isconfigurable such that a control signal received on a first lane of thesecond node controls a future time status change of a second lane of thesecond node. The status of an output lane of the first port may bechanged at a future time determined by the control signal. The futuretime may have a value that is at least one of a fixed time, a timedependent on the position of the node in the interconnect system, or atime value contained in the control signal.

In another aspect, an interconnect system is disclosed, including aplurality of nodes, each node having a port having at least one of aninput link or an output link, each link further comprising a first laneand a second lane. The interconnect system further includes amotherboard having a plurality of connectors, each of the plurality ofconnectors interfacing with one of a plurality of memory modules. Amemory module may communicate with at least one of the plurality ofnodes. Each signal lane has at least one of a signal receiver or asignal transmitter. When a first lane of a first node transmits a firstsignal, and a second node is configured such that the first signalreceived on a first lane of the second node controls a future time powerstatus of a second lane of the second node.

Each node may be configurable such that the control data controls adestination of data received by the node. The destination of the datamay be selected from one of local-to-the-node, or a port.

In another aspect, an interconnect system has a plurality of nodes, andeach node has a port having at least one of an input link or an outputlink, each link further comprising a plurality of lanes and the node maybe configurable such that a control signal may received on a selectedlane of the plurality of lanes. Control data and signal data may bereceived on the same or different lanes. The signal data may be analogdata or digital data. The digital data may be organized as words. A wordis used herein in a generic sense, without restricting the number ofbits in a word, or whether the word is transmitted serially or inparallel, except as specifically stated. Thus a word can be, forexample, any one of a “nibble”. “byte”, full word, double word, wordswith and without parity, and may have non-standard numbers of bits.

In another aspect, an interconnect system has a plurality of nodes, eachnode having a port having at least one of an input link or an outputlink, each link further comprising a first lane and a second lane meansfor controlling the future time status of a second lane based on asignal received by the first lane. The status change may occur betweenstates selected from one of powered on, powered off or standby. Apowered on state may permit at least one of transmission or reception ofthe control or data signal, a standby state has lower power consumptionthan the powered on state, and a powered off state has lower powerconsumption condition, with respect to the standby state.

In yet another aspect, an interconnect system has a plurality of signallanes, and each lane may be capable of at least one of transmitting orreceiving a data signal comprising a bit of a data word. The data wordmay have a first bit and a second bit and each of the first bit and thesecond bit may be logically exchanged between two signal lines such thata differential time delay between the first bit and the second bit maybe altered when the word is received at a distal end of the signallanes. The exchange of bits may be performed at either a proximal or adistal end of the signal lines. Subsequently, the bits may be rearrangedso that the word has the correct logical assignment of data to bits inthe word at the receiving node. Alternatively, the bits may be arrangedat a sending node such that the bits are in the arrangement at areceiving node.

In still another aspect, an interconnect system has a plurality ofnodes, and the nodes form a network connected by links having lanes. Thefirst node may be a source of data, and second node may be a destinationof data, and at least one of the first node or the second node may beconfigured to exchange the logical assignment of data to lanes such thatthe difference in arrival time of data at the second node is changed.

The data may be a word having at least two bits, and a first bit isassigned to a first lane and a second bit is assigned to a second lane.A module controller, may make the logical assignment of bits of a wordto lanes at the input to the first node such that the bits are receivedin a desired order at the second node. Alternatively, the logicalreassignment may be performed at a third node disposed between the firstnode and the second node.

In a further aspect, an interconnect system has a plurality of nodes,the nodes forming a network connected by links having lanes. The firstnode may be a source of signals, and second node may be a destination ofsignals and a means is provided for reducing a differential time delaybetween signals on two or more lanes between the first node and thesecond node.

A node is disclosed, the node having an input port, and an output portand each of the input port and the output port may have a plurality oflines. A switch may be configurable such that each of the plurality ofinput lines are connected to one of output lines, such that a time skewbetween signals on the plurality of output lines may be altered whenmeasured at a second node having an input port having input linescommunicating with the distal end of the output lines from the outputport of the first port.

A memory module is disclosed, having a read-write memory device and aconfigurable switching element (CSE). The CSE may have a first port forat least one of receiving or transmitting signals, and the first portmay have a plurality of signal lanes; The CSE may be configurable tointerpret received control signals to perform functions which mayinclude: changing the state of a signal lane of the plurality of signallanes at a future time; routing received signals to the read-writememory device, or routing the received signals to a second port, asexamples. The state of each of the plurality of signal lines may be oneof powered up, or powered down or standby, as examples.

In another aspect, the future time has a value that is at least one of afixed time, a time value determined by the control signal or aconfigurable value, and may be separately determinable for each of theplurality of lanes.

In another aspect, the CSE of the memory module may be configurable suchlogical data assignment received by the receiving signal lines and thelogical assignment of data to be transmitted by the transmitting signallines is alterable.

The memory module CSE may be configurable such that a signal received onany one signal line of any of the first, the second or the third portsof the memory module may be transmitted on any of the of signal lanes ofthe first, the second or the third ports. The transmitting and receivingsignal lines may be uni-directional, bi-directional or reversible lanes.

In a further aspect, at least one of a data signal, a control signal ora synchronization signal is transmitted on each of the lines or lanes ofa link connecting to a CSE or a memory module during a fixed timeinterval. The fixed time interval may be determined such that at leastone of data edge tracking, clock, clock phase, or frame synchronizationis maintained.

A memory module is disclosed having a means for storing data, a meansfor controlling the future time status of one or more input or outputinterfaces, and a means for routing data to at least one of outputinterfaces or the means for storing data.

In another aspect, the memory module has a means for exchanging alogical and a physical interface assignment for a signal so that adifferential time delay between logical signals on an output interfaceof a first memory module is altered when measured at an input interfaceof a second memory module. In another aspect, the means for exchanging alogical and a physical interface assignment for a signal so that adifferential time delay between logical signals may be altered mayinclude, but is not limited to, fixed arrangements of the electricaltraces on the module, the board(s) the module is attached to, a chippackage substrate, connections, be they physical or proximal, betweenchips in a multichip module alone or in combination with other fixedand/or dynamic means.

In another aspect, the means for exchanging a logical and a physicalinterface assignment for a signal so that a differential time delaybetween logical signals may be altered may include, dynamic switchingfunctions performed both inside a configurable switching element (CSE)as well as by other switching elements disposed between two or moreCSEs, which may include digital switches, analog/RF switches, mechanicalswitches such as microelectromechanical systems (MEMS) devices,configurable proximity communications devices or other similar devices,alone or in combination with other dynamic or fixed means.

A configurable switching element (CSE) is disclosed, having at least twosignal lines, where a first signal line may be configured to receive acontrol signal, and a second signal line may have a controllable state.The state of a second line may be one of powered on, powered off, orstandby, as examples. A control signal received by the first signal linecontrols the state of the second signal line. Address data received onat least one of the signal line, and data received on another signalline may be routed to at least one of another signal line or a memoryinterface. The memory interface is connectable to a means for storingdata.

In another aspect, a configurable switching element (CSE) has a firstinterface for at least one of receiving or transmitting signals, theinterface having a plurality of signal lines, and the CSE isconfigurable to interpret received control signals to perform at leastone of: changing the state of one of the plurality of signal lines ofthe first interface or a second interface, based on a received controlsignal, or routing received signals to another interface, which may be amemory interface, a second interface for receiving or transmittingsignals, or the transmitting aspect of the first interface. The state ofeach of the plurality of signal lines is one of powered up, powereddown, or standby, as examples.

In another aspect, the CSE is configurable such a logical data exchangebetween the receiving signal lines and the logical assignment of data ofto transmitting signal lines is performed. There may be a thirdinterface configurable so that a signal received on any signal line ofany of the first, the second or the third interfaces is transmitted onany of the plurality of signal lines of the first, the second or thethird interfaces.

A configurable switching element (CSE) is disclosed, having a means forcontrolling the future time status of one or more input or outputinterfaces and, a means for routing data to the output interfaces or toa means for storing data.

In another aspect the CSE has a means for exchanging a logical and aphysical interface assignment for a signal so that a differential timedelay between signals on a first output interface and a second outputinterface is altered when measured at an interface of a second CSE.

A interconnection network to accommodate nodes in a network is disclosedhaving connection interfaces with a top end and a bottom end, an mayhave a plurality of signal lines forming first, second and third linkconnections; a network formed by one or more of the first, second orthird links joining the connection interfaces wherein at least two ofthe following configurations of the connection interfaces are joined bytraces:

-   (a) the first through third links being disposed such that all of    the links lead away from the connection interface on a first side;-   (b) the first through third links being disposed such that a link    having a line closest to the top end leads away from the first side    of the connection interface, and the two remaining links lead away    from the second side of the connection interface and the links    leading away from the second side of the connection interface are    disposed adjacently;-   (c) the first through third links being disposed such that a link    having a line closest to the bottom end leads away from the second    side of the connection interface, and the two remaining links lead    away from the first side of the connection interface and the links    leading away from the first side of the connection are disposed    adjacently;-   (d) the first through third links being disposed such that a link    having a line closest to the top end and a link having a line    closest to the bottom end lead away from the first side of the    connection interface, and a link having a line disposed intermediate    between the first two links leads away from the second side of the    connection interface;-   and (e), (f), (g), and (h) where (e), (f), (g) and (h) are (a), (b),    (c), and (d), wherein the top and bottom ends are interchanged by    rotating the pattern of (a), (b), (c) and (d) by 180 degrees about    an axis orthogonal to the connector.

In another aspect, the links may be disposed in layers of a flat printedwiring assembly, or as metallic elements in layers on a substrate. Othercomponents such as memory means, a CSE and other active and passivecircuitry may be fabricated on the same substrate, wiring assembly.

In another aspect, at least three connector interfaces are disposedparallel to each other.

A method of laying out a network interconnect is disclosed, the methodcomprising: disposing a plurality of connection interfaces such that thetops of the connection interfaces are positioned opposite each other andthe bottoms of the connection interfaces are disposed opposite eachother; forming a network of connections between the plurality ofconnection interfaces, where at least two of the following connectioninterface configurations are used:

-   (a) the first through third links being disposed such that all of    the links lead away from the connection interface on a first side;-   (b) the first through third links being disposed such that a link    having a line closest to the top end leads away from the first side    of the connection interface, and the two remaining links lead away    from the second side of the connection interface and the links    leading away from the second side of the connection interface are    disposed adjacently;-   (c) the first through third links being disposed such that a link    having a line closest to the bottom end leads away from the second    side of the connection interface, and the two remaining links lead    away from the first side of the connection interface and the links    leading away from the first side of the connection are disposed    adjacently;-   (d) the first through third links being disposed such that a link    having a line closest to the top end and a link having a line    closest to the bottom end lead away from the first side of the    connection interface, and a link having a line disposed intermediate    between the first two links leads away from the second side of the    connection interface;-   and (e), (f), (g), and (h) where (e), (f), (g) and (h) are (a), (b),    (c), and (d), wherein the top and bottom ends are interchanged by    rotating the pattern of (a), (b), (c) and (d) by 180 degrees about    an axis orthogonal to the connector.

A network controller is disclosed, having a first interfacecommunicating with a computer; a second interface having to a link to anode; and means for determining the differential time delay betweensignals on lanes of a link disposed between a first node and a secondnode.

In another aspect, the network controller may have a means for assigninglogical signals to lanes disposed for communicating between first andsecond nodes so as to alter the differential time delay between thelogical signals received at the second node.

In yet another aspect the network controller may have a computerinterface; a first data interface, the first data interface capable oftransmitting and receiving signals; and, a processor configured togenerate address values for transmitting by the first data interface,the processor further configured to logically assign data values tosignal lines such that a delay differential of logical data on at leasttwo signal lines may be altered when received at a receiving device.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood more fully from the detaileddescription of several example embodiments which follows and from theaccompanying drawings which, however, should not be taken to limit theinvention to the specific embodiments shown, but which are forexplanation and understanding only.

FIG. 1 is a representation of interconnected elements of a network forthe purpose of introducing terminology and symbology to be used in thepresent application;

FIG. 2 is a diagram showing interfaces to a Configurable SwitchingElement (CSE);

FIG. 3 shows a configuration of a CSE which may have more than onesecondary port;

FIG. 4 shows a configuration of a CSE connected in a network with morethan one module controller;

FIG. 5 shows 8 memory modules containing CSEs disposed in a treeconfiguration;

FIG. 6 shows 8 memory modules connected in a binary tree configuration;

FIG. 7 shows another example of 8 memory modules connected in a binarytree configuration;

FIG. 8 shows 9 memory modules connected in a ternary tree configuration;

FIG. 9 shows yet another example of 8 memory modules connected in abinary tree configuration; (where the child modules of M0 are connectedboth to M0 and to the MC)

FIG. 10 shows 8 memory modules connected in a binary tree configurationand additional devices, such as a graphics processor;

FIG. 11 a-f show aspects of a memory system believed to be similar toone being developed by INTEL and known as FB-DIMM;

FIG. 12 a-f show an arrangement of memory modules incorporating CSEs;

FIG. 13 a-f show an arrangement of memory modules having separatecommand and data lines between modules;

FIG. 14 a-f show another arrangement of memory modules having separatecommand and data lines between modules;

FIG. 15 a-g show the transmission of a read command from a memorycontroller (MC) to module Mi;

FIG. 16 shows a module controller and modules arranged in a tree withpoint-to-point links;

FIG. 17 a-d show a configuration similar to the configuration of FIG.16, with separate signaling links used for commands;

FIG. 18 shows an example of a command which may be used to effect therouting using an address indicator;

FIG. 19 shows the action of a switch S where a no-op (NOP) istransmitted from one or more output ports;

FIG. 20 shows a tree of modules where the links are channelized;

FIG. 21 is a further example of a tree of modules where the links arechannelized;

FIG. 22 shows a stylized representation of data transfers;

FIG. 23 a-d show a configuration where many of the links are in a lowpower state;

FIG. 24 a-d show a configuration similar to FIG. 23 where links may beplaced in a lower power state;

FIG. 25 shows a configuration having an extended table of localcommands;

FIG. 26 a-b show the follow of control from the input of S1 to leafs ofa tree;

FIG. 27 a-b show alternate configurations of a self-routing message;

FIG. 28 a-b show a message format which may be suitable for stripping aleading bit;

FIG. 29 a-b show a simplified example of transmitting data from a devicesuch as a module controller to another device;

FIG. 30 shows a method and apparatus for overcoming skew;

FIG. 31 shows a de-multiplexer inserted after the de-skew;

FIG. 32 shows one source of data skew in an INTEL Advanced Memory Buffer(AMB);

FIG. 33 a, b show signal routing to mitigate data skew;

FIG. 34 shows data skew resulting from transmission through 5 modules;

FIG. 35 shows a device and method for reducing differential time delaybetween lanes;

FIG. 36 a-b show an example where the time delays between module pairsare substantially unequal;

FIG. 37 shows an example where lanes 3 and 4 are used as signalingchannels;

FIG. 38 a-e show a pattern of exchanging data elements between lanes;

FIG. 39 shows an example of a Batcher network;

FIG. 40 shows another example of a Batcher network (a-c), and a Closnetwork (d);

FIG. 41 a-b shows an example of a Benes network and an 8 laneconfiguration;

FIG. 42 shows an example of lane exchanging where a full swap is madefor some lanes;

FIG. 43 shows the merging of two data paths at a module;

FIG. 44 shows a clock data recovery (CDR) circuit with phase alignment;

FIG. 45 shows the example of FIG. 44 with de-skew in each of two datapaths;

FIG. 46 a-b, shows the arrangement of FIG. 45 in an more elaboratenetwork;

FIG. 47 shows an aspect of a network having a non-critical data path;

FIG. 48 shows the lanes carrying logical bit 0 in network of 8 modulesconnected in a tree;|

FIG. 49 shows the lanes carrying logical bit N in a network of 8 modulesconnected in the tree of FIG. 48;

FIG. 50 identifies critical and non critical paths in the arrangementshown in FIG. 49;

FIG. 51 illustrates the delays in FIG. 49;

FIG. 52 shows the addition of variable delays;

FIG. 53 shows the addition of delays in a link;

FIG. 54 a-b show equivalent ways of adding delays in a network;

FIG. 55 is a time line showing the delays in the trees shown in FIGS. 46and 49;

FIG. 56 shows another aspect of delays introduced in a network;

FIG. 57 shows the overall times for sending a command and receiving aresponse over a tree;

FIG. 58 a-c shows a situation were the same module may have a differentrouting for transmissions from and to the memory controller;

FIG. 59 shows the tree of FIG. 48 including outbound and return paths;

FIG. 60 a-b shows an arrangement with multiple signaling paths, wherethe signaling paths may be disjoint from the data paths;

FIG. 61 a-c shows a time line for data being transmitted from a moduleto a module controller;

FIG. 62 a-c shows a time line for data being transmitted from a moduleto a module controller using fixed time slots;

FIG. 63 shows a further example of a time line were multiple commandsare being executed;

FIG. 64 shows an example of variable delay scheduling corresponding toFIG. 62;

FIG. 65 shows an example of variable delay scheduling corresponding toFIG. 63;

FIG. 66 a-c shows an example of scheduling where the restriction of datato slotted transmission times has been removed;

FIG. 67 a-c shows another example of scheduling where the restriction ofdata to slotted transmission times has been removed;

FIG. 68 shows the timing for a non-slotted example where the signalingdelays are shown;

FIG. 69 shows the timing for another non-slotted example where thesignaling delays are shown;

FIG. 70 a-c shows the example of FIG. 68 with a different command time;

FIG. 71 shows a further timing diagram with multiple commands;

FIG. 72 shows a tree, similar to that of FIG. 5, laid out in a linearmanner;

FIG. 73 shows the arrangement of FIG. 72 where the modules aredesignated as having a “front” and a “back” side;

FIG. 74 a-d shows an alternative arrangement of modules;

FIG. 75 a-b shows an aspect where the connections between modules may beon other than a motherboard.

FIG. 76 shows a portion of the arrangement of FIG. 72 with alterativemodule connections;

FIG. 77 shows a portion of the arrangement of FIG. 72 with yet anotheralterative module connection;

FIG. 78 shows yet another example of module connections for theconfiguration of FIG. 77;

FIG. 79 shows an arrangement of modules having differing width linkstherebetween;

FIG. 80 shows a grouping of modules in the form of a grid;

FIG. 81 a-b shows a grouping of modules in the form of a hexagon;

FIG. 82 shows a grid comprised of hexagonal structures;

FIG. 83 shows modules in a grid configuration connected to a processor;

FIG. 84 shows a configuration where the processors are connected in atwo-rooted tree;

FIG. 85 a-b shows additional examples of modules connected to multipleprocessors;

FIG. 86 a-b shows yet other examples of modules connected to processors;

FIG. 87 shows the extension of control and data busses into sharedmemory areas;

FIG. 88 shows a configuration of 4 processors and a grid of modulesconfigured as input/output devices;

FIG. 89 shows the configuration of FIG. 88 where the I/O modules arereplaced by other computing system elements;

FIG. 90 shows an arrangement of modules and links capable of supportingboth a linear and/or a tree topology.

FIG. 91 shows the configuration of FIG. 90 used in a linear topology.

FIG. 92 shows the configuration of FIG. 90 used in a tree topology.

FIG. 93 a-b shows a CSE disposed on a module with RAM;

FIG. 94 shows a CSE with RAM chips having separate data and controllines;

FIG. 95 shows a physical configuration of leaf modules with respect tothe remainder of the tree;

FIG. 96 shows an alternative physical configuration of leaf modules;

FIG. 97 shows an arrangement of modules having bi-directional links;

FIG. 98 shows a module with cache memory;

FIG. 99 shows a CSE bridging interconnection systems;

FIG. 100 shows a binary tree of modules disposed in a linear physicalarrangement;

FIG. 101 shows an alternative connection arrangement for the tree ofFIG. 100;

FIG. 102 shows a multi-rooted tree arrangement;

FIG. 103 shows the connection between modules in plan (a) and elevation(b) views;

FIG. 104 shows a multiple-rooted arrangement with redundancy;

FIG. 105 shows another example of a multiple-rooted tree withredundancy;

FIG. 106 illustrates the re-arrangement of the tree of FIG. 105 in thecase of a module failure;

FIG. 107 shows an arrangement having 4 groups of 16 modules;

FIG. 108 shows the air flow arrangement for a group of modules havingfences therebetween;

FIG. 109 shows the air flow arrangement for groups of modules arrangedso that the temperature variation may be controlled;

FIG. 110 shows additional arrangement of heat dissipation elements;

FIG. 111 shows a dual-rooted tree configuration where the nodes havelinks composed of lanes;

FIG. 112 illustrates the layout of traces in a connectorized motherboard of existing design;

FIG. 113 illustrates a layout of traces on a motherboard where theconnectors have been rotated by 30 degrees;

FIG. 114 illustrates that more than one set of lines may be disposedwith minimal bending for the connector arrangement of FIG. 113;

FIG. 115 shows the layout of a larger scale motherboard with theconnector orientation of FIG. 113;

FIG. 116 shows a logical arrangement of 82 modules configured as 64memory modules with RAID error correction and 2 spare modules;

FIG. 117 shows the physical board layout for the arrangement of FIG.116;

FIG. 118 shows an 82 module tree having two roots;

FIG. 119 shows a physical board layout corresponding to a portion of thetree of FIG. 118;

FIG. 120 a-d shows the relationship between the lanes in FIG. 119 andthe layers of a multilayer circuit board;

FIG. 121 shows another tree logical diagram for an 82 modulearrangement;

FIG. 122 shows the physical layout corresponding to the logical tree ofFIG. 121;

FIG. 123 shows yet another logical layout of an 82 module arrangement;

FIG. 124 shows the physical layout corresponding to the logical tree ofFIG. 123; and

FIG. 125 shows the physical attributes of the individual moduleconnector routings which may be used to lay out the physical boardarrangements.

DETAILED DESCRIPTION

Exemplary embodiments may be better understood with reference to thedrawings, but these embodiments are not intended to be of a limitingnature. Like numbered elements in the same or different drawings performequivalent functions. Elements may be either numbered or designated byacronyms, or both, and the choice between the representation is mademerely for clarity, so that an element designated by a numeral, and thesame element designated by an acronym or alphanumeric indicator shouldnot be distinguished on that basis.

It will be appreciated that the methods described and the apparatusshown in the figures may be configured or embodied in machine-executableinstructions, e.g. software, or in hardware, or in a combination ofboth. The instructions can be used to cause a general-purpose computer,a special-purpose processor, such as a DSP or array processor, or thelike, that is programmed with the instructions to perform the operationsdescribed. Alternatively, the operations might be performed by specifichardware components that contain hardwired logic or firmwareinstructions for performing the operations described, or by anycombination of programmed computer components and custom hardwarecomponents, which may include analog circuits.

The methods may be provided, at least in part, as a computer programproduct that may include a machine-readable medium having stored thereoninstructions which may be used to program a computer (or otherelectronic devices) to perform the methods. For the purposes of thisspecification, the terms “machine-readable medium” shall be taken toinclude any medium that is capable of storing or encoding a sequence ofinstructions or data for execution by a computing machine orspecial-purpose hardware and that cause the machine or special purposehardware to perform any one of the methodologies or functions of thepresent invention. The term “machine-readable medium” shall accordinglybe taken include, but not be limited to, solid-state memories, opticaland magnetic disks, magnetic memories, optical memories, and carrierwave signals. The description of a method as being performed by acomputer should not preclude the same method being performed by aperson.

For example, but not by way of limitation, a machine readable medium mayinclude read-only memory (ROM); random access memory (RAM) of all types(e.g., S-RAM, D-RAM); programmable read only memory (PROM);electronically alterable read only memory (EPROM); magnetic randomaccess memory; magnetic disk storage media; flash memory; electrical,optical, acoustical or other forms of propagated signals (e.g., carrierwaves, infrared signals, or digital signals).

Furthermore, it is common in the art to speak of software, in one formor another (e.g., program, procedure, process, application, module,algorithm or logic), as taking an action or causing a result. Suchexpressions are merely a convenient way of saying that execution of thesoftware by a computer or equivalent device causes the processor of thecomputer or the equivalent device to perform an action or a produce aresult, as is well known by persons skilled in the art.

When describing a particular example, the example may include aparticular feature, structure, or characteristic, but every example maynot necessarily include the particular feature, structure orcharacteristic. This should not be taken as a suggestion or implicationthat the features, structure or characteristics of two or more examplesshould not or could not be combined, except when such a combination isexplicitly excluded. When a particular feature, structure, orcharacteristic is described in connection with an example, a personskilled in the art may give effect to such feature, structure orcharacteristic in connection with other examples, whether or notexplicitly described.

A connector or connector interface as described herein, such as a memorymodule connector interface, is not limited to physically separableinterfaces where a male connector or interface engages a femaleconnector or interface. A connector interface also includes any type ofphysical interface or connection, such as an interface where leads,solder balls or connections from a memory module are soldered to acircuit board. For example, in the stacked die approach, a number ofintegrated circuit die (e.g., memory devices and buffer devices) may bestacked on top of one another with a substrate forming the base andinterface to a memory controller or processor through a ball grid arraytype of connector interface. As another example, the memory devices andbuffer device may be interconnected via a flexible tape interconnect andinterface to a memory controller through one of a ball grid array typeconnector interface or a physically separable socket type connectorinterface. Connection types may include the interface between integratedcircuit chips, interconnection conductors on a substrate, betweensubstrates, or on printed circuit boards, or the like.

A memory module may include memory and may also have a controller forthe memory, a configurable switching element (CSE) and other circuitryfor processing, transmitting or receiving signals. However a memorymodule may only include the memory and one or more of the other elementsmay be separately mounted. Except as specifically mentioned, theallocation of the functions to specific modules is intended forconvenience in discussion, as a person of skill in the art willappreciate that the actual physical aspects and computational aspectsmay be arranged in a variety of equivalent ways.

“Bus” or “link” means a signal line or a plurality of signal lines or“lanes”, each having one or more connection points for “transceiving”(i.e., either transmitting, receiving or both). Each connection pointmay connect or couple to a transceiver (i.e., a transmitter-receiver) orone of a single transmitter or receiver circuit. A connection orcoupling is provided electrically, optically, magnetically, by way ofquantum entanglement or equivalents thereof. Other electricalconnections, by the same or similar means are used to provide forsatisfaction of such additional system requirements as power, ground,auxiliary signaling and control, or the like. Such additionalconnections are occasionally described so as to clarify the description,however such additional connections are well known to persons skilled inthe art, and the lack of description of these connections in any exampleshould not be taken to exclude their inclusion.

A link carries signals on a signal lines or lanes. Signals fallgenerally into any of several categories including clock and controlsignals, address signals, command signals, and data signals. Datasignals carry data that may be stored in, or retrieved from, a memorydevice or interface. Address signals specify the location or range oflocations within a memory device or system where data is to be read fromor written to, and may also select which of one or a plurality of memorydevices or interfaces is to be accessed. Command signals instruct amemory device or interface as to what type of operation is to beperformed, e.g., read, write, refresh, and possibly as to which ofseveral access modes (such as a burst or broadcast mode) should be usedfor a data transfer. Clock and control signals synchronize the othersignals passing between controller and the memory devices. Although alink may use a separate signal line for each signal (e.g., 32 addresslines to transfer a 32-bit-wide address in one clock cycle and 32 datalines to transfer a 32-bit-wide data word in one clock cycle), variousschemes also exist to re-use one or more signal lines for differentsignals, various schemes also exist to re-use one or more signal linesfor command signals control signals address signals and data signalsduring different clock cycles of a memory transaction.

It should be appreciated that when lanes or lines are discussed as beingconnected to an interface or to a port, it is convenient to describe thelines or lanes as being active, being powered on or powered off, beingin standby mode, or the like. This is meant to be interpreted asreferring to the status of the connection at an interface with a port ata node. The lane or lines themselves may be merely metallic traces on acircuit board, or the like, serving to provide connections betweennodes.

A “breadth-first” convention is often used in numbering the nodes in atree; that is, to start at a root of the tree and work across all theelements at a given level before moving on to the next level. Thisnumbering is for convenience only. For implementation purposes manydifferent numbering schemes might be used, including two or morenumbering schemes simultaneously. For signaling purposes the convention“port 0, port1, etc” may refer to relative rather than absolute ports.By convention, “southbound”, “downstream” or “secondary” refers to thedirection heading away from a module controller or root while“northbound”, “upstream” or “primary” refers to the direction headingtowards the module controller or root. There may be more that one rootor module controller, and each of them may be operablecontemporaneously.

FIG. 1 is provided to identify some of the symbology and terminology towhich may be used in the examples which are described. This is done forconvenience, clarity and conciseness, and a person skilled in the artwill recognize that a variety of equivalent terminology may be used.However, for the purpose of the description provided herein, the termsset forth here will be used, except when specifically stated. This isnot meant to exclude or distinguish from equivalent accepted terminologyor terminology which may come into use in the future which describeequivalent concepts.

A module controller MC 10 (which may be a stand alone module controlleror memory controller, or may be an interface or circuitry within someother device such as a central processing unit (CPU), digital signalprocessor (DSP), microprocessor, graphics co-processor, bridge chip, orany such device as might need to be connected with an interconnectsystem, but which is not shown in the figure), whose function may residein one component or may be spread across more than one device, isconnected to a group of modules M#, 20 (or other computer systemcomponents such as an interface, another processor, memory, or the like)by point-to-point links. In the description the term “module” or “memorymodule” may be used to represent other computer system components aswell, including interfaces and processors.

There may be as well one or more multi drop busses for control andsignaling lines; there may also be one or more global clocks, and boththe signaling lines and the data lines may be uni-directional orbi-directional. The links between the module controller MC 10 and themodules (M0,M1,M2,M3), and from one module to the next, may contain bothdata and control/signaling lines. The links may be configured in manydifferent ways, and some possible configurations are shown in FIG. 1.

Groups of lines or lanes shown where:

-   -   L0 depicts separate data 30 and control lines 40, where each set        of lines has separate northbound and southbound unidirectional        lines of equal width;    -   L1 depicts separate northbound and southbound unidirectional        data lines 30 a, b of unequal width, northbound control lines 40        a from M1 to M0 and southbound control lines 40 b from MC to M1        and northbound data lines 30 c from M1 to MC;    -   L2 depicts north- and southbound bi-directional data 30 d and        control lines 40 c between M1 and M2 as well as separate        bi-directional control lines 40 d between M1 and M2, also        southbound uni-directional control lines 40 e from M1 to M3; and    -   L3 depicts combined northbound uni-directional data 30 e and        control lines 40 f from M3 to M2, bi-directional southbound        control lines 40 g between MC and M3 and uni-directional        southbound and northbound data lines 30 f, g from M2 to M3.    -   Control lines skipping adjacent modules such as the control        lines from MC to M1 (40 b), M1 to M3 (40 e) may be used to send        control signals with lower latency than if sent through each        hop, similarly with data lines such as from M1 to MC (30 c).        Lines labeled “data”, or “control”, may be only “mostly” data or        control lines, and they may also be used to carry signals of the        other types. That is, control signals may be carried on the data        lines and data signals carried on the control lines. As such,        data and control lines may be logical or physical, depending on        the context.    -   Clock signals may be distributed to the various components and        modules directly from a clock generation circuit, using a        single-ended clock line or a differential clock connected        between two or more modules. A high-speed clock may also be        combined with data or control signals.

FIG. 2 shows an example of a Configurable Switching Element 15 (CSE),which may have more than one secondary, or downstream, port. The CSE maybe used to communicate with memory or other devices; the memory or otherdevices may be located on the same physical module as the CSE or may belocated on a separate module from the CSE, on the mother board, or atanother location. Two arrows associated with a link indicate abi-directional data path, which may be separate uni-directional links,or bi-directional links, or may be logically bi-directional connectionsmade by running uni-directional links in a ring-like fashion. Links mayhave a serial or parallel configuration, or be a combination of seriesand parallel configurations and be either single ended or differential.

The CSE may have the capability to connect any input port to any outputport, without limitation. For convenience in logical description, theports may be considered northbound or southbound in the presentdescription, however such a description does not serve to limit thecapability of the ports of a CSE to communicate to each other. For,example a northbound port may communicate with a southbound port withina CSE, or a southbound port may operate as a northbound port in amulti-rooted tree connection. Of course, not all of these capabilitiesmay be configured in a particular arrangement.

FIG. 3 shows another configuration of a CSE 15 which may have more thanone secondary, or downstream, port and may also have more than oneupstream, or primary, port. In addition, the CSE may have one or moreinternal memories in addition to or in place of external memory and theexternal connections may be with devices other than memory modules orsystems, such as I/O devices, co-processors and the like. The CSE mayhave multiple secondary ports, as shown, so that the CSE may be used asto split the signal into more than one path. Where a single set ofnorthbound and southbound ports is provided, the CSE may act as arepeater. The terms “internal” and “external”, when used to refer tomemory should be understood to be convenient for descriptive purposes,but not limiting. Memory circuits are increasingly being incorporatedinto microprocessors or special purpose hardware, and the use of termssuch as internal and external may be used to assist the reader inunderstanding a particular example. A person skilled in the art willunderstand that the combination or separation of circuits andcapabilities into differing physical components, such as beingintegrated in the same package or on the same substrate, and the like,is governed by specific applications and the state of the associatedtechnology.

FIG. 4 shows another example of a Configurable Switching Element (CSE),which may be connected in a system with more than one module controllerand may not have an “upstream” or a “downstream” direction of data orcontrol flow. The CSE may have various internal processing functions (orbe incorporated inside or operate in conjunction with devices havingvarious processing functions) such as, microprocessors, direct-memoryaccess (DMA) engines, the CSE itself being a module controller forcontrolling other CSEs, and the CSE may be externally connected todevices other than memory systems such as input/output (I/O) devices,microprocessors, graphics processors, co-processors, other CSEs, etc.The use of the terms “primary” and “secondary” are thus seen to be usedfor convenience in description. In the situation where a CSE contains amicroprocessor (or is contained within, or coupled to, amicroprocessor), the CSE may act as a processing unit as well as aswitch.

FIG. 5 shows 6 memory modules containing CSEs having associated memory,disposed in a tree configuration with a memory controller MC at a rootthereof. For convenience, the CSE is not shown and the CSE that iscontained in memory module M# may be described as CSE#. A memory modulemay contain both memory and a CSE; however, some memory modules may onlycontain a CSE. In the present configuration, memory module M0 isconnected to memory modules M1 and M2 and, for example, memory module M1is connected to M3 and subsequently to M7. In such a configuration thenumber of hops to the furthest module is 4 (M0→M1→M3→M7) when comparedwith 8 hops if the 8 memory modules were arranged in a linear ordaisy-chain fashion. A hop is considered to be the link or laneconnecting two successive nodes in a network. Hops may be used as adescriptor when time delay or differences in time delay of thepropagation of signals between nodes of a network are being discussed.

FIG. 6 depicts 8 memory modules connected in a binary tree configurationsuch as may be disposed on a PC board in a physically linear arrangementof modules. Note that some of the links such as between M0 and M1 and M2and M6 pass by intervening modules M4 and M5 without necessarilyconnecting thereto. This aspect is illustrated by shading the link whereit bypasses a module. The arrangement of FIG. 6 shows that a largenumber of tree elements such as memory or other modules may be arrangedon a circuit board, or as plug-in modules to a mother board in anorganized fashion, such as a linear physical arrangement. Such anarrangement may facilitate the layout of a mother board or in coolingthe modules. Combinations of linear and non-linear arrangements may alsobe used effectively dispose modules on a board or mother board whosedimensions are constrained for other reasons, such as the dimensions ofthe rack or equipment in which the board is to be mounted.

Mother boards may be used to facilitate the repair of a memory system byreplacing failed memory modules. Whether a mother board is desirable maydepend on failure rate and mean-time-to-repair (MTTR) analysis and theprobability of irretrievable data loss, among other factors.

Many of the examples discuss the arrangement of elements as being atree, however this is not meant to limit the topology of theconnections. Examples are shown where there is more than one entry pointor “root” and commands and data may flow along non-“tree” paths, such asin a circular fashion. The tree examples often permit a clearerdescription of the situation, and may simplify description of suchaspects as reverse path scheduling. However, this is not meant topreclude the use of contention resolution, queuing and store and forwardtechniques, as examples. That is, the arrangement may be generalized toother network configurations. In addition, more than one independentmemory controller may be employed or controllers may operate in acoordinated fashion.

It should be understood that while a specific number of memory modulesis shown in the figures, and specific numbers of modules are discussed,this is not meant to limit the number of memory modules or other deviceswhich may be connected in such a manner, and is merely for conveniencein presenting drawings to facilitate description of examples.

FIG. 7 depicts 6 memory modules connected in a binary tree configurationsuch as may be disposed on a PC board in a non-linear arrangement. Suchan arrangement may be convenient based on the physical requirements of aspecific design.

FIG. 8 depicts 7 memory modules connected in a ternary treeconfiguration such as may be disposed on a PC board in a non-lineararrangement. This illustrates the use of a CSE with more than twosecondary ports to configure a system having a ternary architecture, aswell as where not all of the possible nodes of a tree are populated withdevices.

For simplicity, the examples presented will generally limited to binarytree structures, however the arrangement of FIG. 8 illustrates thatternerary, quaternery and higher order trees may be constructed. Theterm “tree” is meant to encompass trees of arbitrary order. A linear ordaisy-chain arrangement of some or all of the modules is not excludedwhen the term tree is used without qualification, and the use of theterm binary tree means a tree of binary or higher order. As mentionedpreviously, the use of a tree as an example is not meant to limit theuse of the apparatus or method to the examples, but such apparatus andmethods may be applied to networks generally.

FIG. 9 depicts 8 memory modules, connected in what is essentially abinary tree configuration, which may be disposed on a PC board in anon-linear arrangement, where the memory controller MC is also connectedto the modules in the second level of the tree (M1 and M2). Thisconfiguration may be used to reduce the latency of communicationsbetween the MC and the memory modules as well as to reduce the powerconsumption of M0 by diverting some signals from passing through M0.Such a configuration may also serve to facilitate increased throughputby issuing commands, and receiving replies as well as sending orreceiving data over connection links or lanes from MC to the secondlevel of the tree (M1 and M2) as well as through the connections to thefirst level (M0).

FIG. 10 depicts 8 memory modules connected in a binary treeconfiguration which may be disposed on a PC board in a non-linearphysical arrangement with other devices or modules such as a graphicsprocessor (GPU) device connected to M4 and a network I/O deviceconnected to M5. Each of the other devices or modules may haveassociated internal or external memories, GM and NM, respectively, whichcan be accessed directly and which the MC can also access through theCSE on each such module. Providing direct access to a memory module GMfrom a GPU may further improve the overall throughput by permittingmultiple access operations to occur without conflict.

For comparison purposes, a baseline configuration is shown in FIGS. 11a-11 f, intended to represent aspects of a memory system believed to besimilar to that being developed by INTEL, which is known as afully-buffered DIMM (FB-DIMM). The description herein is based on aninterpretation of documents made available publicly be INTEL; but, asdetailed descriptions and specifications do not appear to have been madegenerally available, no representation is made that the devices shown inFIG. 11, as produced by INTEL, actually operate as described herein inan INTEL configured memory system. Rather, where a comparison with abaseline architecture is convenient, the arrangement described in FIG.11 is meant. Included in each memory module m (200) is a device havingthe interface characteristics as have been disclosed by INTEL for theINTEL advanced memory buffer (AMB).

The signaling convention for a read command and response utilized hereindiffers from that which appears to be used by INTEL in order to simplifythe functional description. For example, in the INTEL AMB, a “read”command takes several messages, which have been compressed fordiscussion herein into one message. Additionally, the AMB appears tohave a capability of sending a single “frame” containing commands formore than one AMB so, for example, for some commands one can not assertthat a “frame” is meant to be received by a particular module in variousstages of passage between a memory controller mc (300) and a memorymodule mk, each of the memory modules having an advanced memory bufferAMB.

In FIG. 113 a, the read command is issued by the memory controller (300)as an address Ai and a read command R and transmitted over thedownstream path as a “packet”. In the example given, the data addresslocation is in memory module mi. FIG. 11 b shows the packet arriving atmemory module mi (200 i). However in a FB-DIMM system, the packet isrepeated promptly from one module to another down the linear chain.Thus, even though the read command was addressed to memory module mi, asshown in FIG. 11 c, the packet is further forwarded to memory module mk(200 k). Therefore, each of the packets containing the read commandtraverses the full length of a chain of memory modules 200. The responsefrom memory module mi (200 i) represents the data read from address Aiand includes data bits, or data packets, d0-d3 in this example is shownin FIG. 11 d, traversing the upstream path. The number of data bitsshown is limited for convenience in presentation and may be more orless, or may be variable. FIGS. 11 e and 11 f show the data passingthrough memory module mO and being received by the memory controller mC.The packet of read data is forwarded by all modules by a process ofrepeating without being de-skewed, de-serialized, or decoded.

In an example, FIGS. 12 a-12 f show a series of memory modules M#,organized in a linear fashion for comparison with the baseline shown inFIG. 11; however the memory modules of FIG. 12 incorporate CSEs which“snoop” or read the address contained in the command packet even whenthe packet is not intended for the specific CSE or its associatedmemory. Each CSE may be in a state where the upstream paths are in a“reduced” power setting, shown as a dashed line. Reduced power mayinclude but is not limited to, de-powering the I/O drivers, gating theclock of the I/O logic (and any other associated logic that can bestopped), reducing the clock rate of the I/O logic, reducing the voltageof the I/O logic, loading preset inputs to the I/O logic that aredesigned to reduce the leakage current of the I/O logic, or any othermethod of for reducing the power consumed by any portion of the chipwhich may be undone quickly enough to enable the handling of thereturned data. In the example shown in FIGS. 12 d-12 f, the upstreamlinks power up in advance of the returning data and then return to areduced power state after the data passes.

In this example, the control signals travel as a packet over the datalines where the downstream path is powered up. In an aspect, where thepackets are transmitted in a “framed” or “slotted” timing system, thedownstream path may be powered up at the beginning of each “frame” or“slot” time and if there is a packet to be sent the transmit side mayremain powered up and the packet sent; otherwise the transmit side maybe powered down until the beginning of the next slot time, and thereceive side will be powered up until a determination is made as towhether a packet to be received; if there is no packet, the receive sidemay power down until the start of the next slot time.

Specifically, FIG. 12 a illustrates the situation where a MC (10) issuesa read command R for an address Ak. In FIG. 12 b, the read command Rdata packet arrives at memory module Mi (20 i), for which it is notintended, and passes through to memory module Mk (FIG. 12 c). As aresult of the data read command R, a packet of data D0-D3 is transmittedupstream (FIG. 12 d), passing through memory module Mi (FIG. 12 e) andbeing received by the memory controller MC (FIG. 12 f). In the sequenceof FIGS. 12 c-f, the powering up of each successive upstream link priorto transmitting the read data over the link is illustrated, as well asthe powering down of each link after the passage of the read data D0-D3.In the present arrangement, the downstream links are all powered up, asthe control signals as well as data travel in data lines of thedownstream path.

As the data read command R packet passes along the downstream path fromM0 to Mi and to Mk, each of the memory modules M observes or “snoops”the data read packet and ascertains both the destination module and thespecific nature of the command: in this example, to read data from anaddress in Mk. Such read data traffic is expected to flow upstream fromMk to the MC. Consequently, each of the links in the upstream pathbetween the module from which the data will be read and the MC may bepowered on at an appropriate future time (shown as a transition from adashed line to a solid line) to transmit the read data, and may bereturned to a low power state or turned off when the read data has beentransmitted over the link. Further, in the event that the read datacommand R is addressed to a memory module other than the last one in achain, only the upstream links for the path between the addressed memorymodule (e.g. Mi) and the MC need be powered on at appropriate futuretimes, resulting in a reduction of power consumption.

FIGS. 13 a-13-f depicts a configuration of memory modules M employingCSEs having separate command and data lines between modules. The signaland command lines may be merely a subset of the data lines rather thandedicated signal lines as shown. In the configuration shown, someportion of the downstream links and their associated logic may be putinto a reduced power state. As the command control signal passes eachmodule the signal is decoded and, if appropriate, other downstream linksmay be powered up to transmit the data or command which follows in adata packet. In the aspect shown, a read command R is issued for anaddress Ai in memory module Mi, where read command R and the addressdata Ai are sent on separate lines. The address data Ai indicates thatthe desired address or address range is in memory module Mi. As theaddress data packet Ai is transmitted earlier than the read command R,the address data packet Ai at each of the memory modules M# earlier thanthe read command R, as may be seen in FIG. 13 a-b, and the address datamay be used to power up the link between the receiving module Mi+1 andthe transmitting module Mi so as to accommodate the transmission andprocessing of an expected command. Similarly to the situation describedin respect to FIG. 12, the downstream command path may be powered downagain after the command has been received, the upstream links may betimely activated for the transmission of the data read as a result ofthe command.

Thus, the power status of one line or lane may be said to be alterableat a future time, based on the control or command signal, address ordata signal being received by a CSE. The time value of a future timestatus change may be determined by a characteristic of the receivedcommand, the address of the destination, the corresponding position ofthe receiving CSE in the network, or similar or derived information, ormay be intended to occur promptly for some or all of the lines or lanesof a CSE.

In another aspect, shown by a comparison of FIG. 13 b and FIG. 13 c, thecommand signal may be altered by the module (in this example Mi) that isthe target of the command so that a no operation (“no op”) command maybe sent on the downstream lines to indicate to the modules that arefurther downstream that no command will be sent to them in the next“frame,” “cycle” or whatever the time period that may be used as aninterval between commands. As an example, the read command is absent inFIG. 13 c, however the address data Ai is received by Mk, which isfurther down the chain that the addressed module. This may not result inactivation of the downstream command link between Mi and Mk. Thesequence of data return shown in FIG. 13 d-f follows that of FIG. 12 d-fand will not be described in detail.

Each of address data packets may be decoded as it passes. through amemory module (actually the CSE 15 of the memory module 20), whereas thecommand packet may merely be passed along, and the additional delay inthe address data packet results in a decrease in the time intervalbetween the arrival of the address data packet and the command packet asthe signals progress further downstream. In an aspect where the addresspacket and the command packet are transmitted sequentially on the samedata lines, a time interval is left between the trailing end of thefirst packet and the leading end of the second data packet in order toaccommodate the additional processing delays encountered by the firstpacket. This may be desirable in some situations, but does represent aloss of bandwidth with respect to the illustrated example.

In a further aspect, the address data may be simultaneously decoded at amemory module and also transmitted downstream, so that the decodinglatency may not be experienced. In some instances this may result in acommand being sent one link further than actually needed, but dependingon the configuration, the aspect of the CSE relating to receiving thecommand at the extra link end may not be activated.

In another aspect, FIG. 14 a-f illustrates arrangements similar to thoseof FIGS. 12 and 13. But, in the previous examples, a capability wasprovided for each module to determine whether the module being addressedwas upstream or downstream of the present module. This determination mayrequire the comparison of the received address against a large number ofmodule addresses and, depending on the complexity of the logic beingused, several clock cycles to complete. This processing time may resultin an increase of the separation required in the time between thetransmission of the address and the command by the MC. This additionalseparation may be mitigated by modifying the address Ai, received by thedestination module Mi, prior to transmitting the address data to thenext downstream module Mk, so that a new address AX is a flag toindicate that the address and the command has already been interpretedby the appropriate module M. AX may be a reserved address or somesimilar indication.

As a result, each module may only have to compare the address Aireceived against the address or range of addresses of the module Miitself, or the reserved address, rather than against all of the moduleaddresses in the system. In other respects, the example of FIG. 14 issimilar to that of FIGS. 12 and 13 and a detailed repetition of thedescription is omitted.

High-speed data links may use a clock recovery mechanism to maintainsynchronization. If a sufficient number of bit transitions per unit timeare not received, the clock time synchronization of the link may belost, and the recovery of synchronization may take some time, which mayreduce throughput, or cause data loss which may result inretransmissions. In order to maintain synchronization, systems such asFB-DIMM may periodically transmit synchronization (“sync”) frames whichcontain such transitions, and are sent to all of the modules.

However, in configuration having a large number of memory modules, suchsync frames would be sent to each of the modules, and therefore traverseall of the links in the system, or multiple sync frames could be sent tothe various leaves of the tree. If a tree of modules is sufficientlylarge, this process for maintaining time synchronization may consume asignificant portion of the bandwidth of the system, and the need forsynchronization of clocks may tend to limit the size and complexity ofthe system arrangement unless clocks are refreshed, or the clocksynchronization is maintained by other means.

An alternative approach is to provide each of the modules with a timer,to keep track of the time interval elapsed since there have been asufficient number of transitions and to initiate clock resynchronizationon a local basis. An example of this is a counter or a watch-dog timer.

FIG. 15 illustrates the transmission of a read command from the MC tomodule M0. The read command R is forwarded to all of the modules, sothat the command lanes are not de-powered. Under the condition that apredetermined time interval has elapsed, and the lane is not being usedfor transmitting data, the module initiates the transmission of data,which may be a “no op” data packet to continue to maintain thesynchronization of the link. FIG. 15 d shows an example where module M0has determined that no packet will be using the M0→Mi link in a frameand uses the opportunity to transmit transitioning signals on the lanesof the link for the purpose of maintaining the receiver synchronizationin module Mi. Likewise Mk has determined that the Mk→Mi link will not beused in that frame and transmits transitioning signals to Mi.

Further, if the link is half-duplex, or full duplex, the link or lanesor lines thereof may be turned around so that the transmission is on thepath between the modules not having a high traffic density. Such anapproach may depend on the reception of command information so that themodule can determine that no data is expected from the other directionand turning around of all or a portion of the half-duplex link may beperformed. This is shown in FIGS. 15 e-f where modules Mi and Mk reversethe Mk→Mi link in FIG. 15 e and then, in FIG. 15 f, Mi sendstransitioning signals to maintain the receiver sync in Mk. It isworthwhile reiterating that the arrow headed lines represent links,where the arrow head indicates the current direction of data flow, thata link may be one or more signal lines (lanes) for data, command andaddress data, and that direction of a link may be capable of beingreversed from that shown in a particular figure, as is the situationwhere a link between Mi and Mk has been reversed in FIG. 15 f from thesituation in FIG. 15 g. In this discussion, the capability of beingreversed in transmission direction may be presumed to be possible butnot necessary, as actual implementation may be application dependent.

FIG. 16 illustrates a module controller MC and modules M0, . . . , M6arranged in a tree architecture, having point-to-point links. Such atree may be operated where data packets containing an address areforwarded and each switch (such as a CSE) or module (such as a memorymodule or other module incorporating a CSE) decodes the address anddetermines where the corresponding data or command should be directed.However such processing takes time at each step: to receive the data, tocompare it with destination addresses, and to determine the action to betaken. When the address is sent at a time in advance of the command ordata, and a serial link is used, the time interval between the addressand the following data will decrease at each hop due to the dataprocessing required for the address evaluation. Such action takes longerthan merely forwarding the data without processing. Performing the dataprocessing functions in parallel may require additional complexity,including forwarding data to links which are not desired. Whetherparallel processing is performed depends on the overall networkbandwidth desired.

An additional consideration is the time dispersal of the data bits ineach data packet. In many links, the bits of the data may be sent onparallel signal lines or lanes. It should be understood that this may asimplification of the situation, as the data may be sent over parallellines with more than one bit transmitted on each line in a serialfashion, as an example. Due to differential delays of the data on thelines the differential delays associated with, amongst other factors,line length and impedance, the data bits may not arrive with sufficientsimultaneity for immediate processing and it may be necessary to waitfor one or more clock cycles until all of the bits have been received bya module in order to proceed with decoding or other data processing.This delay further increases the time necessary at each node before theaddress or other information can be acted upon. In such a circumstance,and where it is desired to turn on the lanes or lines of links only asthey are required for data transmission, the address information mayhave to be sent a significant period of time in advance of the commandor the data. This has the effect of reducing the system transfercapacity or bandwidth.

FIG. 16 has similarities to FIG. 12, but the modules are arranged in atree and only the relevant paths are shown. The MC issues a read commandR for an address in module M5 (FIG. 16 a). At M1, the address and dataare routed to M2 while the paths to M1 and modules associated therewithare not used (but “no op” commands may be sent as will be laterdescribed) (FIG. 16 b). At M2, another decision is made by the local CSEand the address A5 and read command R are forwarded to M5 (FIG. 16 c).Comparison of the spacing between the address packet A5 and the readcommand R, as shown in FIGS. 16 a-c shows in a qualitative manner theprogressive reduction in time interval between the two packets, suchthat in FIG. 16 c where the address and command data packets are beingreceived at M5, the time interval between them may be minimal. Thiswould correspond to a situation where the interval between the issuanceof the two packets at the MC was set such that substantially all of theinterval would be eroded by the decoding delays at intervening modules.The response of module M5 to the read command R(FIG. 16 d) follows thatpreviously described and is thus not described in detail here.

FIGS. 17 a-d illustrate a configuration that is similar to that shown inFIG. 16, but separate signaling lanes are used for the commands, inaddition to the previously described address or data lanes. As there maybe separate lanes for the commands and the address or data, it may bepossible to more efficiently utilize the bandwidth of the address anddata links. The propagation delay of the address may be presumed, forsimplicity of description in this example, to be the same between eachof the modules, so that the transmission of the read command can beadjusted to result in a desired interval between the address and dataarrival time at the destination module (in this example M5). This may beadvantageous where a series of read commands R are issued in succession.

As with the previous examples, the address link and the command link maybe powered up at an appropriate future time in advance of the use, andpowered down after the information packets have traversed the link. Itwill be appreciated by persons skilled in the art that some efficiencymay result from permitting each of the links to remain powered up for aperiod of time after use to accommodate closely spaced packets ofinformation. Such a period of time may be characterized in terms of amultiple of clock cycles.

FIG. 18 illustrates one example of a command which may be used to effectthe routing using an address indicator. The topology of this figure isthe same as that of FIG. 18, although, for clarity, only the commandlines are shown, and the possibility that the links are turned on andoff as needed has been suppressed. In this example, the address commandis structured as a self-routing packet rather than a destinationaddress, and contains three two-bit words followed by a local command tobe executed at the destination module. The address words are shown asenclosed in thin hexagons and the local command is shown as beingenclosed in bold hexagons. Further, each of the words may becharacterized as having a least significant bit (LSB) and a mostsignificant bit (MSB) for ease of description. The four two-bit wordstransmitted to S0 are shown as if they were transmitted on two parallellines. Alternatively, the information may be transmitted on a path overa signal line in bit serial order.

The interpretation of the bits in each word is shown in the tableaccompanying FIG. 18. Note that, for generality, each of the modules isshown as S0-Sn, to suggest that each of the nodes may be a switch only,or in addition to other electronics being addressed at the switchlocation. A switch may be a CSE or other device capable of interpretingand responding to the commands.

Each of the switch modules Sn, in this example, has an input port, andtwo output ports, the output ports being designated as “0” and “1”. Theaddress command is interpreted in accordance with the destination selecttable, where the value of the two-bit word determines the output port tobe used. Although only two output ports are shown, this is not meant tolimit the number of output ports of the switch element, or the number ofbits in the each address command word. Further, each of the input andoutput ports and the upstream and downstream directions are used forconvenience in discussion and a connection between any logical orphysical input port and any output port is not precluded.

The final word in the address packet contains a local command. That is,an operation to be executed at the destination switch. Examples ofcommands are given in the local command table. For example, a value of 1(binary 01) selects the CSE, a value of 2 (binary 10) selects the localmemory, and a value of 3 (binary 11) is a no-operation command (NOP or“no op”). FIG. 18 a shows the address command packet at the input to S0for a command intended for S5. The leading bit pair (binary 10; MSB,LSB) results in the selection of output port 1 of S0. FIG. 18 b showsthe data at the output of S0, where it will be noted that the leadingword 1 (binary 10) has been stripped form the address command, as it hasalready been acted upon at S0. In this manner, each switch S maydetermine a course of action based on the leading word in the packet asreceived. However, the position in the packet of the word that is to beinterpreted by each switch S may be other than the leading word. Inaddition to simplifying the processing, the time to interpret theaddress command in this format is the same at each of the modules Sn, sothat the incremental delay to perform this operation is independent ofthe depth of destination in the tree to which the address command isdirected.

After passing through switch S2, and being routed to port 0, as theleading two-bit word is 1 (binary 01), the leading word is againstripped away. FIG. 18 c shows the address command that will be receivedby S5. The leading word is now 3 (binary 11), local, indicating that thedata following is intended for the local module. The next following wordis interpreted as a command to be executed locally, in this case 3(binary 11), a NOP. Any response from the addressed module istransmitted back upstream.

In another example, shown in FIG. 19, the action of a switch S may besuch that a NOP is transmitted from one or more of the output ports notselected by the address command. This may facilitate the propagation ofclock pulses throughout the tree in an efficient manner to maintainsynchronization. The address command at the input to S0 is againintended for S5 with a local memory command 2 (binary 01). As seen inFIG. 19 b, when compared with the original command shown in FIG. 19 a,the address command appears at port 1 of S0 with the leading wordstripped away. As such, port 0 was not selected and therefore may issueapacket to the next switch connected to port 0 (that is, the port of S0connected to S1) consisting of a local address, followed by a NOP. FIG.18 c depicts the action of S1 in response to the receipt of the NOP.Since neither of the ports of S1 was selected for data output, theaction of S1 may be to issue packets at both port 0 and port 1consisting of a local address, followed by a NOP, which propagate to S3and S4. The action of S2 may be different, and the input address commandas shown in FIG. 19 c may result in the routing of the address commandto port 0 (binary 01), stripping away the leading word, and transmittingthe remainder of the command packet to S5. But, port 1 of S2 was notaddressed in this situation, and a packet containing a local address anda NOP to S6 may be output. Each of the modules at the end of the tree(“leaves” or “leafs”) (in this example S3, S4, S5, S6) may also issue alocal command and a NOP at each of the output ports thereof, however, ifno further paths are connected, the process terminates. Modules may alsobe configured to act as tree leaves and the downstream output portsthereof may either be inactive or omitted.

The tree of modules may be configured such that the links can bechannelized as in FIG. 20, using a CSE 15 of the type shown in FIG. 3,and permitting the contemporaneous or simultaneous execution of two ormore commands, depending on the degree of sub-division. In this example,two separate data packets are illustrated. A first packet which mayresult from an execution of a read command R is shown returning frommodule M4 to the MC via M1 and M0. At the same time a second packet,which may result from the execution of a read command R is shownreturning from module M6, through M2 and M0 to the MC. The returningpackets may use a number of lanes of the line, and the number of lanesmay be less than the full number of lanes in the link between modules.It is not required that the division of lanes between the “channels” ina link be identical.

A further channelization of the tree is shown in FIG. 21. Here, at thesame time as a read command R is being processed from each of modules M3and M6, a direct memory access (DMA) transfer may be conducted betweenmodules M4 and M5 via M1, M0 and M2. The transmission of data frommodule M4 to M5 may occur at the same rate as, and without interruptionof, the transfers in progress from M3 and M6. In such a configurationDMA transfers may be performed to or from any module not being accessed,or from any module capable of processing more than one data path at atime. In this way, direct memory access may be performed in thebackground without affecting the performance of other accesses occurringessentially simultaneously. With arbitration within the module, or wherethe module can support more than one operation/access at a time, such aswhere a module contains a multi-ported memory, or where the CSE canbuffer some or all of a conflicting operation, it may be possible toperform operations sent from the module controller to modules which aresimultaneously performing operations internal to the tree such as DMAs.DMA operation is also illustrative of the connection of a downstreamport to an upstream port as an example of the possibility of connectingeither logical or physical ports within a CSE to effect the datatransfer.

FIG. 22 is a stylized representation of data transfers occurringcontemporaneously or simultaneously in, for example, a treearchitecture. In this example a read operation from module M8 (shown asa dashed gray line between module M8 and the MC), passing through M4, M1and M0, and a write operation from the MC to module M9 (shown as a solidgray line between the MC and module M9), passing through modules M0, M2and M5. Data corresponding to a read command R is being transmitted frommodule M5 to M4 (shown as a dotted line) passing through modules M2, M0and M1. Thus, it may also be possible, depending on the specific sourceand destination addresses, for accesses to be performednon-simultaneously from the MC while memory transfers are occurringwithin the tree of modules.

FIG. 23 depicts a configuration where many of the links are in a lowpower state (shown by dashed lines). When the reception of an addresscommand indicates that data will be following in the downstreamdirection, in this case a read command R, those downstream paths overwhich the read command R will travel are powered up so that the commandmay be transmitted. This is shown in FIG. 23 b by the change of the MCto M0 link to a solid line. FIG. 23 c shows the powering down of the MCto M0 link and the powering up of the M0 to M2 and M2 to M5 links. FIG.23 d shows the subsequent powering up, at an appropriate time, of the M5to M2 and M2 to M0 links to transmit the data being returned in responseto the read command R.

FIG. 24 depicts a configuration similar to that of FIG. 23, except thatthe upstream paths may be placed in a lower power state. Generally areturn to normal operation for a path is longer when the path circuitryis in a very low power consumption state. The upstream data paths areshown as a dash-dot line when in such a state. (The distinction may bemade between, for example, a powered-on state, a standby state and apowered-off state, or the like. In making these distinctions, adifference in the overall power consumption and a difference in the timeto transition to a full performance state is implied, but the amount oftime or the amount of power will be specific to a particular design).Aspreviously discussed, each of the modules on the path between the MC andthe destination module, in this case M5, becomes aware (by “snooping”)that a specific command type has passed through each module (in thiscase M0 and M2), and the depth in the tree to which the command wasaddressed. Therefore it may be possible to schedule the re-powering ofeach upstream link at a future time such that it may be in operational(powered up) status when the data is being transmitted over the link.FIG. 24 a-c also illustrates the transmission of NOP commands N tobranches of the tree that are not on the address path.

In a system where commands may be constrained as to when they may betransmitted, for example in a system with slotted transmission time, orwhere there may be a minimum interval between commands, or a restrictionon when certain commands may be sent, further power savings may beachieved. FIGS. 24 a-d show how, for example, in a system with slottedcommand timing, the link carrying the address command might be poweredin time for the beginning of the slot, as shown by the half dashedlines. If there is no command to transmit, then the link may be powereddown until the beginning of the next slot time. If there is a command totransmit, then the link remains powered up so the command may betransmitted. FIGS. 24 b-d also show how in a system without fixedcommand start times, a NOP command N may be sent to the links not beingtaken by the command to permit the unused links to power down theassociated command link until the time at which another command may needto be received, whereupon they are powered up again.

For certain physical lane transmission technologies, the powering up ordown of a lane may require action by both the transmitter as well as thereceiver ends of the lane.

As a further example of an address command, an extension of the commandmay be made to facilitate broadcast or multi-cast distribution ofcommands. FIG. 25 has an expanded table of local commands, including abroadcast command having a value of 0 (binary 00), and a table ofextended commands. The word associated with the extended command isshown in the hexagon formed of dotted lines. Although shown as the lastword in an address packet, the position of the extended commands may bedifferent. The extended commands include Read having a value of 1(binary 01), Write having a value of 2 (binary 10), Broadcast-starthaving a value of 3 (binary 11) and Broadcast-forward having a value of0 (binary 00). The extended command word may follow the local commandword in the transmitted sequence. In this example, an address command isdirected to S1, as the first word in the address selects port 0, whichis connected to S1. A NOP is transmitted from S1, port 1, which is thenon-selected port, and propagates ultimately to S5 and S6 as shown inFIG. 25 b, c. With respect to the packet sent to S1 (FIG. 25 b), thefirst word indicates that S1 is the destination of the command, and thatthe local command has a value of 0 (binary 00), corresponding to abroadcast command. Switch S1 may create new messages in response to thebroadcast command. A broadcast command may be issued from each of port 0and port 1 of switch S1, being addressed to S3 and S4, which are thenext switches in the tree, connected to port 0 and port 1, respectively.The broadcast command has an extended command of broadcast-forward,having a value of 0 (binary 00) as the extended command. As such, S3 andS4 may generate similar commands which propagate down the tree.

FIG. 26 a, b shows a detail of the flow of control from the input of S1to the leafs of the tree. Such a configuration may be used forsimultaneously sending and/or receiving small individual status/controlmessages to/from many modules.

FIG. 27 a illustrates some alternate configurations of a self-routingmessage. In an aspect where very high speeds may be desired, the skewfrom lane-to-lane may be such that only messages transmitted on a singlelane may be interpreted without the significant delay which may beexperienced in waiting for the other portions of the message which weretransmitted over a second or greater number of lanes. Here, a number ofbit-serial formats are described that demonstrate various propertieswhich may be useful.

It may be desirable that the first switch route the message with minimaltime delay. As an example, there is the special case of a messagedestined for the first module in the root of the tree. Here, the firsttwo bits in the message (reading from right to left) are binary 01, andthe command CMD (which may be a read R, write W, or other command)follows. In an example where the message may be directed to the lowestlevel N (for example a leaf), the first two bits have the values binary11. This is the followed by the path indication [PATH], which may be Nbits of path information, where the bit pattern is, for example, 0 or 1,depending on the port to be used at the output of each successive branchor node in the path to the destination. This path indication is followedby the command CMD being addressed to the specific module. A messagewhich is destined for the first from the bottom level has the leadingbits 011 and then a path indicator of length N−1. The next highest levelhas a leading bit pattern 001 and a path indicator of length N−2. Forthe N−3 level, the leading bits are 010, indicating that furtherinformation as to the depth of the message follows. Then the patternbinary 11 marks the beginning of the path indicator. In this manner thelength of the leading bit pattern grows by approximately one bit forreach level in the tree, but the path indicator

decreases in length by one bit for every level, and thus the totalnumber of bits in the preamble and the path indicator may be constant towithin nominally one bit. As such, the message length has littlevariation in length associated with the depth of the module to beaddressed. FIG. 27 b illustrates preambles for messages to be sent toS9, S4, S1 and S0 of the arrangement of modules shown in FIG. 26.

Also shown in FIG. 27 a are additional commands CMD-TIME and CMD-EXwhich contain information about when the command should be executedand/or data returned. The time may be absolute time or relative timefrom the receipt of the command, or may have an interpretation based onthe nature of the command itself.

FIG. 28 a shows a message format which may be suitable for stripping theleading bit at each node. The format for an address to module 0 would bea leading 0 and then a bit that indicates whether a command follows or aNOP. A message addressed to level 1 is a leading 1, followed by a portselector bit P, indicating that the port 0 or port 1. The pattern ON,again indicates whether a command for the module follows (1), or a NOP(0) is desired (0). For addresses of modules further down the path, apair of bits is added for each level, indicating the added depth androuting on the path. At each level, the first two bits of the messagesmay be interpreted to determine if the message had been addressed to thelocal switch or is to be forwarded. If the message is for furtherdownstream in the tree, the first two bits of the message are strippedoff and the message is forwarded via the appropriate port. The numbersin the rectangular boxes at the right hand side of the figure representthe depth or rank in the tree.

FIG. 28 b illustrates the progress of a message being transmitted to S9at various stages down the tree, where the leading bit pairs arestripped off after they have served their function of routing themessage.

As bus (link) speeds increase, the problems associated withsynchronizing the data with the clock or commands in a synchronous busarchitecture may increase. One of the problems is that separate endpoints of a link, lane or line may have different clocks. Also, ifparallel data transmission techniques are used, the effective physicallengths of the individual lines in the links may generally be different,and there may be a time difference in the arrival of bits sent ondifferent transmission lanes or lines. This may be termed data “skew”.One method of addressing the problem of data skew is to perform de-skewat each hop such that the header of the packet may be interpreted andthe packet transmitted on the next hop. However, this may addsignificant latency time to the re-transmission as the de-skew processrequires at least as much time as the longest differential delay betweenlines between each node. Other approaches attempt to avoid thisadditional delay by not performing de-skew at each node in the tree, butperform de-skew at the receiver of the addressed module or node, whileallowing the skew to build up along the path. But the cumulative delaycan also add significant total delay, as the differential delayaccumulated from the source (e.g., a memory controller) to thedestination node must be accommodated, even though part of the datapacket may have arrived significantly earlier.

In the FB-DIMM technology as being developed by INTEL, devices arebelieved to be required to be able to correct for a skew of up to six UIper hop. The term “UI” or “Unit Interval”, which is the average timeinterval between transitions of a signal, may be used at high clockrates in place of a clock period, as the clock period may not becompletely stable or free of jitter. If six UI of skew per hop isexpected, and a memory system has 8 hops, the total skew is 48 UI.

In the case of a link running at a clock speed of nominally 2 GHz with adata rate of 4 Gbps, 1 UI=250 ps so a delay skew of 48UI is theequivalent of 12 nanoseconds (ns) of skew. That is, it is assumed that12 ns is needed between the first arriving bit on one lane and the lastarriving bit on one of the other lanes. The first arriving bits may needto be stored in memories, latches or shift registers until such time asthe later arriving bits arrive, and then the bits are presented to thedevice which is to de-skew and read the data.

In an aspect, the interconnection system described herein may provide amethod and apparatus for operating in such askew environment even wherethe delay from device-to-device along a link is not fixed, in such amanner as to mitigate the need for delay equalization, delay scheduling,buffering or the like. FIG. 29 a illustrates a simplified example oftransmitting data from a device such as a module controller MC toanother device, which may be a switch S, a memory module M, aninterface, another processor, or the like. In this example a wide word,for example 16 bits, is converted by a serializer SER (300) into four4-bit words. That is, the 16-bit word 310 is spread across four separatelanes and transmitted at four sequential high speed clock times. Thedata is received at the other end of the link, de-serialized by thede-serializer DES 320 and reformatted as the originally input word.

FIG. 29 b shows the same operation, but in a situation where the bits donot all travel along the links at the same speed and arrive more thanone clock cycle out of step with each other. The cause of thisdifferential delay may be, for example, differing line lengths betweenthe modules. As shown, bits 1, 3, 5 and 7 arrive earlier than theremaining bits and, unless time delay skew is accounted for, the bitsmay be interpreted as being part of the preceding transmitted word. Theremaining bits would be interpreted as being in the next word, and thedata would be corrupted.

In an aspect, FIG. 30 illustrates a method and apparatus for overcomingthe skew. A switching element with clock data recovery (CDR) receivesthe skewed data. Alternatively, the CDR may not be used. De-skewconsists of delaying each lane by a time such that the delays in alllanes are equalized. The de-skewed data is then applied to ade-serializer (DES) and the originally transmitted word is recovered,and used by the CSE or other local function. Although the individualfunctions are shown separately, some or all of the functions describedmay be performed in the same electronic device, or may be integrated ina module. Nor is it necessary for de-skew and the CDR to be performed inthe same circuit. Further, as shown in FIG. 31, de-multiplexers 330 maybe inserted between the output of the CDR and de-skew 340, and the DESand transmitter TX (350) such that the data input may be sent to the DES(320) or TX (350), or both. Such a configuration permits acting on adetermination as to where the message should sent such that the data naybe sent to the DES or the TX as appropriate, rather than being processedby all of the circuits.

Among the causes of skew is the situation shown in FIG. 32 where anINTEL Advanced Memory Buffer (AMB), or the like, is in a module which isdisposed on an extender board that may be inserted into a connector.Such arrangements are used for FB-DIMM, registered DIMM, and the like.The message on the upstream path is shown coming from a downstreammodule, having a number of lanes, where lane 0 is on the left, lane 4 isin the middle, and so on. The upstream path interfaces with a front sideof the connector and, after passing through the AMB, connects to asimilar arrangement of lanes on the back side of the connector. Notethat while the AMB is shown on a front side and back side of the plug inboard, separate AMB elements are not required to perform this function.Additionally while the lanes are shown only to the edge of the AMB theymay connect underneath the AMB if the AMB is in a ball grid array (BGA)package or other package which allows connections not only at the edgethereof. The lanes emerge from the AMB to a connector on the back sideof the board. The lanes on the back side of the board are also disposedsuch that lane 0 is on the left and lane 4 is in the center, and so on.The lines or lanes shown may be considered to represent traces on theboard for connecting between the AMB and the board interface connector(not shown). The lanes are disposed as shown so that more than onemodule can be connected by a motherboard with traces connecting themating module interface connectors without a need for crossing traces.In this aspect, the time for signals to propagate between the pins ofsuccessive connectors on a mother board may be approximately the samefor each lane. However, this is not the situation on the module board.Lane 4 has a short distance to travel from the connector to the AMB,whereas, for example, lane 0 has a larger distance.

For illustrative purposes, a standard size DIMM module is used as areference. Such modules are about 140 mm in width, and therefore, thedistance to lane 0 from the center, where the switch is located, isabout 70 mm. Attributing a length of 20 mm to the length of lane 4between the AMB and a corresponding pin on the connector, the maximumdifferential length, in this example, between the shortest and longesttraces is about 50 mm, which corresponds to an approximate difference inpropagation time of about 250-350 ps. This represents the estimatedsituation with an existing DIMM. Where the data rate is nominally 4GB/s, this may result in one or two clock periods of skew from thiseffect alone.

While several standardized DIMM or memory modules exist, the term DIMMor memory module as used herein should be understood to refer to any ofthe memory types which may be used as mounted to a circuit board havinga connector interface for connection to other modules circuits and thelike. The number of pins or traces, the dimensions of the circuit board,and the capacity and type or types of memory on the DIMM or memorare notrestricted to such pin counts interconnect counts, dimensions, types andcapacities as are now being used.

FIGS. 33 a, b show how different approaches to signal routing may beused to mitigate the differential propagation time problem. FIG. 33 ashows a situation where the traces are fanned out directly from the CSEto the connector. FIG. 33 b shows a situation where the signal lineswith longer traces are routed on the top of the board and then anotherset of signal lines are routed from the middle. In such a circumstance,the propagation distances for some traces are greater than in FIG. 33 a,however the difference in trace length between groups of signals, suchas between 0,1,2 and 3, or between 4,5,6 and 7 can be reduced. Whilethere may still be a difference between in length between groups oftraces, the process of delay management and signal de-skew may besimplified by reducing the delay variation from, N different delays forN signals to 2 different delays in the example of FIG. 33 b.

FIG. 34 shows an example of transmission from a module controller MC,shown at the left had side of the figure through five modules, for lanes0-4. The data being transmitted from the MC has been designated as A, B,C, D, and E corresponding to the lanes. At each of the downstreammodules a table shows the logical name of the input “I”, the delay “D”experienced by the data on that lane, the total delay “T” from the MC tothe present position along the data transmission path and the logicalname of the output data “O”. The rows of the table correspond to theorder of the lanes; thus the upper row shows lane 0 and the bottom rowshows lane 4. In this example, a delay is associated with each lane, andthe delay may be different for each lane. The delay may be interpretedas a differential delay with respect to a notional lane with no excessdelay or of the delay from the transmitter on the previous module, orthe MC, to the receiver on the receiving module.

For this example, where the delay between modules is presumed to rangefrom 1-6 (arbitrary units) for illustrative purposes, and the input andthe output lanes from each module has the same data assigned at theoutput of a given lane number as that which arrived on the input, theoutput data pattern associated with each module is the same as the inputdata pattern: A, B, C, D, and E associated with lanes 0-4 respectively.By following the progression of the tables from module-to-module in thedownstream direction, the total delay for each lane is the sum of thedelays for that lane between the MC and the module for which isassociated with the table. For example, the total delay experiencedbetween the MC and the 5th module (M4) is 30 units for lane 0, but only5 units for lane 4. The other lanes, in this example, experienceintermediate amounts of time delay. If the delay units were associatedwith clock cycles, then the data on lane 0 would arrive 25 clock cyclesbehind that for lane 4. The delays are given in integer values and whichare the same in a given lane between each of the modules are forillustrative purposes only. The delays need not be integer valued normust they be the same at each hop between modules for a given lane. Inthis example, if the total length of the command, or packet, or frame,being sent was, for example, 10 clock cycles, then about 2.5 commandswould be in the process of being received at any time by module M4. Thusthe last module may need to buffer more than two complete commands atany time before it could process the first command, as all of the bitsof the first command will take this long to be received as a completegroup. In this example, the second command and part of the third commandwill have begun to have been received at module M4 before all of thebits of the first command will have been received.

FIG. 35 illustrates a device and method for reducing the differentialdelay between the lanes, so that the accumulation of skew in a systemmay be mitigated, and the number of commands that may need to becontemporaneously processed at each node or hop may be reduced. In thisdevice, the input data to each module may be output to a different dataline from the input data line, with respect to the layout on the plug-inmodule or the circuit board or substrate. The actual physicalarrangement of the lanes is the same as in FIG. 34 for purposes of theexample of FIG. 35. However it should be understood that using AMB or aCSE may result in differing numbers and types of links and signal lines.The example is meant to illustrate the differences between performanceof the existing AMB, as previously described, in a data skew sense, andthe method and apparatus of the present application. In some instances,the input and output data lines have the same logical assignment aspreviously, so that only some of the data may exit on a different lane.

For example, at the first module (M0), the data A that came in on lane 0is output on lane 4; the data B that came in on lane 1 is output on lane3; the data D that came in on lane 3 is output on lane 1; the data Ethat came in on lane 4 is output on lane 0; and, the data C that came inon lane 2 is output on lane 2. At the output of module M1, thedifferential delay situation is the same as shown for module M1 in FIG.34. However, when considering the table for module M2, the situation haschanged. As an example, the data A, which is now on lane 4 for thetransit between modules M1 and M2, has only accumulated the delayassociated with lane 4, which is 1 unit; and, the data E, which is nowon lane 0, accumulated the delay associated with lane 0. At the secondmodule M2, it is seen that the delay for the data E is 7 units and thedelay for the data A is also 7 units. Delays for the other data B, C, Dare not exactly the same, but are close in value, and the re-assignmentof logical and physical data lines has reduced the spread in delays.

Following the reassignment or exchanging of the data A-E to lanes 0-4 ateach module, the difference in delay between the various logical dataassignments A-E is seen to accumulate monotonically, as in the previousexample. The data may continue to be exchanged at each module. At thelast module (in this example M4), the associated table shows that therange of delays associated with the various logical data elements rangesfrom 15 units to 18 units, for a total delay spread of 3. This may becompared with the example of FIG. 34, where the delay spread was 25.Interchanging the correspondence between the logical data assignmentsand the physical lanes thus may reduce the amount of data which may needto be buffered before all of the data for a given clock period at thetransmitter has been received at a module by about a factor of 8. Inthis example, the bits are reordered in module M4 into the same logicalorder as were transmitted.

The bit assignment strategy used in the example was to select the datawhich had experienced the highest delay and assign it as an output tothe lane having the lowest delay. Similarly, the data with the lowestdelay is assigned to the lane with the highest accumulated delay.

In the description, the delays are given integers values for simplicityof presentation; however non-integer values are also possible as thedelay is stated in arbitrary units representing some factor multiple ofthe clock rate. Alternatively, the delay and differential delay may beexpressed in integer and fractional clock durations.

FIG. 36 illustrates an example where the delays between module pairs aresubstantially unequal. Only two lanes are shown, however thisarrangement may be generalized to any number of lanes. The two lanesshown may represent lanes on which commands are transmitted. FIG. 36 aillustrates a situation similar to that shown in FIG. 35, where thelanes are swapped such that the difference in delay is minimized atevery hop along the path. A swap is made at the first and third modules,and no swap is made at the second and fourth module. The lanes wouldhave been reordered, if it had been necessary, at the fifth module. But,the delay in lane 0 between the fourth and fifth module is 10 units,whereas the delay in lane 0 is 1 unit. As the differential delay hadbeen minimized along the path, the large difference in delay at the lasthop becomes evident in the total differential delay at the fifth module,where the total delay for lane 0 is 18 and the total delay for lane 1 is9 units, and the difference between then is 9 units. FIG. 36 b shows analternate situation, where the differential delays are permitted toaccumulate along the path until the last but one module, and then thelanes are interchanged, and the interchange reversed at the last module.At the last module the differential delay is zero; however, it should beobserved that as the differential delay between the fourth and the fifthmodule is substantial, a significant differential delay has been allowedto build up prior to module where the interchange is performed so thatthe differential delay after the interchange is performed is reduced.

The delays are shown as having been previously determined, as the basisfor the setting of the lane swapping switches. Such a determination maybe made at the time of system design or prototyping, the time ofmanufacture, the time of assembly of the modules into a system, systemstart up, each system start up, or by measurements made during theoperation of the system. The selection of the time or times to performthis determination may be associated with the desired performance andthe complexity of the system, and may take into account manufacturingvariability and temperature effects as well as active or passivecomponent failure or degradation.

The function of interchanging exchanging or “swapping” lanes may beperformed by a switch, however the use of the term “switch” does notserve to limit the specific implementation of the function to a discretedevice; any mechanism that achieves the interchanging, exchanging,re-assignment or swapping of the lanes may be used. For convenience theterm “switch” when applied to this aspect of the system or method ismeant to encompass the range of techniques which are available toperform the function. Such techniques may use memory storage locations,operation of a state machine or a stored program computer, and dedicatedelectronic logic circuits and the like, either alone on in combination.Such switching may be performed on a static or a dynamic basis,depending on the design requirements of a specific product.

FIG. 37 shows an example where lanes 3 and 4 are used as signalingchannels, and were chosen as the lanes having the lowest overall delay,whereas lanes 0-2 are allocated to data signals. The signaling channelsare de-skewed by interchanging such that the propagation delay isshorter than that of the data channels so as to have the command arriveat the same time or earlier than the data. The data channels arede-skewed by one of the methods previously described. While this exampleconfines the signaling lanes to lanes 3 and 4, the signal lanes may beswapped into any of the lanes such that the commands arrive at the sametime as the data at the destination module. Moreover, some of the datamay be routed over lanes having a shorter delay than being used for thesignal on a particular hop, so long as the overall differentialtransmission time is such that the data and command signal are receivedin a timely manner. Doing this may result in minimizing the overalldelay time for the data signal.

FIG. 38 a shows a pattern of exchanging the data elements between thelanes in order to minimize the differential data delay at any module.The tables of delays are not shown, but are presumed to have led to thepattern of swapping that is shown. At M4, the data arrive in a swappedorder C, D, E, A, B, whereas the expected order is A, B, C, D, E andthis order is restored by swapping the data appropriately inside moduleM4 such that the data is presented to the remainder of the system in theexpected order. FIG. 38 b summarizes the overall effect of the swappingas seen from the output of MC to the input of M4 and within M4. That is,the intermediate hops, and the swapping that occurs at each hop may belogically replaced by a single swapping of lanes from MC to M4 and aswapping of lanes inside M4. This represents the end-to-endtransformation that obtains. The configuration of the switches at eachmodule is as shown in FIG. 38 a, but the ordering of the data at the MCis such that the same result is achieved at the input to M4. In anotheraspect, shown in FIG. 38 c, the switch arrangement remains unchanged butthe order of the assignment of the data to the lanes is pre-permuted atthe MC prior to transmission such that it is received at M4 in theexpected sequence and no reordering is required in M4. Such an approachmay minimize the delay in processing at each module, and when theprocess is performed in the MC, the pre-permutation may be done whilethe command and data are still in an output queue and the time toperform the operations may be subsumed in other system processes. Suchpre-permutation may also to simplify the data path within the CSE byallowing incoming data to either be permuted for transmission to thenext module or to be read without permutation for use by the local CSE.If the CSEs have been arranged in a branching configuration such as atree, then a large fraction of the CSEs in the system may be leafs andhave no other CSEs downstream from the leafs. This pre-permutation maybe made in a static or dynamic manner.

The situations illustrated are directed to a MC to module transfer, butmay be effectively used for other paths, such as DMA transfers betweenmodules, and for different channels or branches of a tree. While thediscussion has addressed the path from a MC, for example to a module,these operations may be equally performed in the reverse direction.

FIGS. 39-42 illustrate various examples of switches and switchingmethods which may reassign logical data to lanes. In FIG. 39 a a Batchernetwork configuration is shown in having four switch elements forconnecting lanes numbered 1-4 at the input to lanes 1-4 at the output.FIG. 39 b, the configuration of a 2×2 switch in such a network is shown,where the switch configuration is capable of two positions: in one ofthe positions, the lanes are passed through so that the input lane andthe output lane are connected to the same lane number; in the otherposition, the switch interchanges the lanes between the input and theoutput. FIG. 39 c illustrates a switch which, in addition to thefunctionality shown in FIG. 39 b, also includes a copy function, wherethe input of lane 1 may be duplicated to the output lanes, or the inputof lane 2 duplicated to the output lanes. FIGS. 39 c and d show theswitches in electronic schematic form where the circuit element is a 2to 1 multiplexer. In FIG. 39 d, the configuration of FIG. 39 b isrealized and in FIG. 39 e, that of FIG. 39 c is realized. A differencebetween the two circuits is that in the case of the 2×2 switch withoutcopy function, the multiplexer select lines are activated in common,with one of the multiplexers having a NOT input 420. In the 2×2 switchwith copy function, the individual switches are separately controlled.

FIG. 40 a illustrates a Batcher network being used to swap input lanesdesignated 1, 2, 3, 4 so that the ordering is 3, 2, 1, 4 (the output isshown with the output lane number first and the number in parenthesisshows which input lane is being transmitted over that output lane).However if an attempt to make an output assignment of 1, 3, 4, 2 wasdesired, FIG. 40 b shows that a path would be blocked.

FIG. 40 c shows a non-blocking crossbar-switch matrix. This type ofswitch may be used to perform any necessary swap. Another type of switchis a Clos network, shown in FIG. 40 d, which is of a type known asstrictly non-blocking, wherein it is possible to change the orderingbetween the input and output lanes, while only changing those laneswhose assignments are actually changed.

The capability of the Batcher network switch may extended by addinganother stage as shown in FIG. 413 a, and this is sometimes termed aBenes network. Now, the swapping that was desired in FIG. 40 b ispossible. However, the additional stage of switching may introduceadditional delay. Nevertheless, this is a reconfigurable non-blockingswitch. FIG. 41 b shows the extension of such a switch to an 8 laneconfiguration. The breadth of the switch is not limited.

In some arrangements, it may be found that one or more patterns of laneinterchange are frequently encountered. FIG. 42 illustrates a situationwhere a full swap is a common requirement for some lanes of a system,and is shown with a device a circuit element connecting between, forexample input lane 1 and the multiplexer that outputs to output lane 8.The other inputs and outputs are connected similarly. Also shown are twoBenes networks connecting the inputs and inputs to the outputmultiplexers such that other arrangements may also be configured. Thepermutations desired are then selected by the output multiplexers incombination with the fixed network and variable configuration switches.While only one fixed network is shown, more than one fixed network maybe used with multiplexers having greater input capacity, such as a 3 to1 multiplexer. As may be done for many of the circuits being described,the circuits not being used may be powered down or powered off, or havetheir clocks, if any, gated, or any other steps which might be desirablytaken to minimize power consumption.

Systems may have variable delays in the timing on a path or in acircuit. The timing variations may result from thermal expansion orparameter variation, or other deterministic or non-deterministiceffects. Whenever there are variable delays in a system, the signalpropagation time from the root (such as a MC) of a tree to a module isnot fixed at a determined value. Small-scale phase variations occur,even if the clocking is such that there may not be a need to correct forphase variations and the data can be latched from a global clock wherethe clock speed is sufficiently slow that the data is not skewed; but,there are nonetheless different propagation distances from device todevice, or from any one device to the head end (root). Theseconsiderations still apply as the clocking becomes less perfectlysynchronous, where there is skewing of the data or jitter in the globalclock, or where there is a phase difference with respect to the globalclock and the links are self timed.

FIG. 43 illustrates the merging of two data paths, which may arrive at amodule, for example, from two separate branches (links) of a tree. Atthis point the differential delay properties of each tree may be suchthat a differential delay exists between the branches at this node, andwithin each branch, between the lanes. The functions being described maybe performed in a CSE, or in associated circuits, and the presentationof any of the functions is not intended to illustrate their physicalallocation, but to illustrate the function performed in each element.

The data on the four lines of data path are shown as numbers, and thedata on the four lines of data path II are shown as letters, for clarityin discussion only. With respect to data path I, the data are shown asbeing well aligned in time between the channels. The data enter a clockdata recovery unit (CDR). A separate CDR may be used for each lane, orfor groups of lanes. The data of path II is shown as having a time skewbetween the lanes. The output of each CDR is input to a multiplexer 500,which is configured to select one of the two groups of input data andoutput the data to the transmitter TX (510) or other circuit. The fourlines connecting between the multiplexer and the TX show both the path Iand the path II data, although the effect of the multiplexer is toselect only one of the paths. The two lanes are shown after themultiplexer only so that the timing differences may be easilyappreciated; however only one of the paths is selected and actuallypresent at the location between the multiplexer and the transmitter TXfor a particular multiplexer selection. A comparison of the datapresented shows is that there would be a misalignment between the dataof path I and path II with respect to a clock, even if a separate clockis used for each data line. For clarity of discussion, a single clock ispresumed for each of the two data paths, although a separate clock maybe used to each of the lanes of each of the paths.

When the data of either path I or path II has been selected, andtransmitted, the effect on the next CSE in a string of CSEs, or any datareceiver needs to be considered. For discussion purposes, the next CSEin a string is termed the upstream CSE. At the upstream CSE, which maybe configured to be the same as the CSE shown in FIG. 43, the receiveddata will now represent a single data path, on which either the numberedor alpha data has been placed by the multiplexer and transmitted by thetransmitter TX. This may be the path I input to the upstream CSE, whileanother branch of the tree may be the path II input. The followingdescription applies to either the path I or path II inputs, but will beexplained with respect to the path I input, whose data came from the CSEshown in FIG. 43.

The CDR may have aligned the recovered clock with that of the path I(numeric) data. When the multiplexer switches to path II (alpha), theclock rate associated with this data may be different, and so thesampling of the data by the clock rate at the upstream CSE may result inerrors.

FIG. 44 illustrates a circuit which may mitigate the situation, at leastin part. The CDR has a phase alignment circuit which adjusts the timedelay of the data such that data from data path I and data path II arealigned with a clock edge. This may be seen by comparison of the inputdata on paths I and II and that of the multiplexed data at the output ofthe transmitter TX. Recall that only one of the two paths will actuallyappear at the output of the transmitter TX at any time, but the two setsof data are shown to illustrate the relationship of the two data sets toa common clock. While this alignment may overcome some of thedifferential delay between merging paths, the data on line 2 shows thatthe data of the two paths is offset by a full clock period, in thisexample. In fact, this offset may be more than one clock period; and,the skew still remains.

FIG. 45 illustrates the situation in FIG. 44 with the addition of ade-skew process in each of path I and path II. The de-skew be achievedby adding the equivalent of one clock cycle delay in the data of lane 2of path II, and the result is that the data of path I and path II forlane 2 can be selected without a change in the apparent skew of the dataas received by an upstream CSE.

FIG. 46 a illustrates the concept of FIG. 45 on a higher level in anarchitecture. A memory controller MC and modules M0-M7 are connected ina tree. The time delay of each data path is represented by the length ofthe line connecting modules Mi, and the time delay within each module isrepresented by the length of each module. From FIG. 46 a, it may be seenthat the path MC-M7 is the longest path. Only one of the lanes betweeneach circuit element is shown of the K possible lanes, but thedescription is equally applicable to the other lines or lanes. In FIG.46 b, the tree is redrawn so that the horizontal axis represents timedelay for a lane connecting each of the circuit elements, including thedelay in each circuit element. For each data bit, the lane betweencircuit elements being used may be a different physical lane, but thelogical data bit is correct at the circuit element where the data orcommand is interpreted. That is, the set of lines shown which eventuallyleads to lane 0 at the MC are those lanes which carry the data whichwill arrive on lane 0 even if those physical lanes are of a differentlane number between other parts of the tree, for example as a result ofthe of lane as exchanges shown in FIGS. 39-42

FIG. 47 a identifies two of the sub-trees of the network: M0, M2, M6 andM0, M1, M3, M7. As shown, the path M7-MC is the longest path. At eachcircuit element, two input paths and one output path are shown. One ofthe input paths is non-critical in the sense that the non-critical pathhas a shorter overall delay between the origin and the destination thanthe other path. For example, at module M1, the path from M4 isnon-critical with respect to the path traversing M7-M3-M1. In thiscircumstance, an additional delay may be introduced into the M4-M1-M0path with respect to the M7-M3-M1-M0 path without introducing additionaloverall delay. When an additional delay is introduced, this is shownsymbolically by showing a tab 550 extending from the module where themodule connects to a data path.

Using the symbology introduced in FIG. 47, FIG. 48 shows a network of 8modules Mi, connected in a tree to a module controller MC, for thelogical bit 0 path. The time delay for the transit of the bit from themodule Mi to the MC for bit 0 is represented as dM_i_N, where i is themodule number and N is the logical bit number. The delay is measuredfrom the output of the module Mi to the input of the MC. That is, delaysin the originating module Mi and the receiving circuit MC are not shownfor simplicity. It should again be mentioned that the additional delaysin a module, and shown by the tab on a module, may occur anywhere in themodule. When considering the timing in the network, the time tM_i_Nrepresents the time at which a command or data for logical bit 0,transmitted from the MC will be received at module i. FIG. 48 shows thesituation for module 5 and bit 0, so that the time of receipt tM_5_(—0=t+dM)_5_0, where t=0 is the time when the data was transmitted bythe memory controller MC.

A similar analysis may be performed for any bit N, and FIG. 49 a, b showthe same arrangement of modules as in FIG. 46, b, except that the delaysare appropriate for another bit, bit N. As shown, in FIG. 50 a, b, theremay exist a different grouping of sub-trees having maximum path time forbit N and thus at each module combining paths there may be a differentset of sub-trees having non-critical paths. Here, the sub-trees with themaximum time delay are M5, M2, M0, and M7, M3, M1, M0, MC. The first subtree is different when compared with the bit 0 situation previouslydescribed, but the second sub-tree has not changed. Thus, for bit N, thedelay may be added at the port interconnecting M2 and M6, whereas in thecase of bit 0, the additional delay may be added at the port connectingM2 and M5 (as shown in FIG. 47).

FIG. 51 shows the resultant delays represented in the notationpreviously introduced. Thermal and other slowly time varying effects mayrequire adjustment of the delays to maintain the phase and bitsynchronization of data which is being selected by a multiplexer in amodule and sampled by a common clock. The trapezoidal extensions 560 ofthe delays shown on modules M0, M1 and M1 in FIG. 52 indicate suchvarying additive delays, where the total length 565 of the extensionindicates the maximum total additive delay, and the dashed transverseline 570 indicates the minimum additive delay. In another aspect, due toa number of factors, which may include internal construction of amodule, processing delays, and the like, the delay within a module maynot be the same for each lane. When different lanes are chosen for alogical bit transmission between modules, the module delay may thusdepend on the lane chosen. This is represented by a box with a broad tab575 as shown in FIG. 53. Again this is a symbolic representation anddoes imply that the delay is necessarily introduced at the input or theoutput of the module.

Depending on the method of time delay control chosen, it may be possibleto introduce the delays for skew correction at a module prior to that atwhich de-skew is needed. FIG. 54 a, b show examples of two means ofintroducing the delay. As previously shown, in FIG. 54 a delay is shownbeing added to M2 in the connection to M6, where the path M5 to M2 isnon-critical. Alternatively, a delay may be added in M5 and the delayeliminated in both M2 and M0 as shown in FIG. 54 b. This may be seen tohave reduced the overall delay between M6 and M2 and the MC, withoutdegrading the delay between M5 and the NC.

FIG. 55 a shows the delays for the trees described in FIGS. 46 and 49,where each of the delays is shown for the bits designated 0 and N. FIG.55 b shows the same trees, but with module M7 deleted. A long delay wasremoved from the configuration, to show that, as represented in FIG. 55b, there is no “longest delay module”. A longest delay for bit 0 is seento occur on the path to module M6, whereas a longest delay for bit N isseen to occur on the path to M4.

FIGS. 46-55 show examples of the time delay, for example, on a returnpath between a module M and a module controller MC. The transmission ofcommands and data between the MC and the modules M, and the return of anacknowledgement or data also involves time delays; however, thesituation may differ from the return path in an aspect that the returnpath requires multiplexing of data from more than one path, whereas thetransmission of a command or data from the MC may be performed in amanner where the data fans out downstream in the tree by branching.While phase correction may be needed for synchronization, skewcorrection by the addition of time delay may not be required. So, thetime delays on a path from the MC to a module M and the time delay on areturn path for the module M may not be the same, even for the same bit.In FIG. 56, while there may be delays introduced at a module, forsimplicity of presentation, all of the delay between modules, includingany skew of phase synchronization delay is shown as part of the lengthof the line collecting the modules. FIG. 56 a shows a tree where acommand has been transmitted from the MC, and the total delay ininter-module transmission is shown as a dotted line. In accordance withthe discussion on path management, the command, if not a broadcastcommand, will not actually traverse each of the connections betweenmodules, but may traverse only a subset of the links depending on thedestination address. But to the extent that it traverses a particularpath, the delay in doing so is shown.

FIG. 56 b shows a tree such as has been previously shown in FIG. 48. Thelines between the modules are the sum of the lines shown in FIG. 48 andthe lines in FIG. 56 a. The dotted lines 580 represent the time delayencountered by bit 0 transiting from the MC to each of the modules, andthe solid lines and the tabs represent the delay in a return signal fromthe module to the MC for bit 0. For purposes of presentation, the timeto prepare and transmit signal from the MC, the time to decode and actupon the signal at the addressed module and to transmit a responsesignal, and the time to decode and act upon the signal at the MC areomitted.

FIG. 56 b therefore shows the total transit time of a command from theMC for bit 0 as sent to any one or more of the modules Mi, and theresponse of Mi to the command as it is received by the MC.

FIG. 57 shows the situation of FIG. 56 b in greater detail, introducingnotation to represent the overall transit time of a command and aresponse over the tree. sdM_i_N represents the delay between thetransmission of a command (or data) from MC to Mi to the return of thefirst bit on lane N from the module Mi to the input of the MC.stM_i_N(t) is the time from when a bit N is transmitted at time to thetime that a response is received at the transmitting circuit. This isshown for module M5 where stM_5_0(t)=t+sdM_5_0 for the situation where acommand has been sent from the MC as bit 0, to module M5 and a responsehas been received at the MC.

The previous examples have shown situations where the topologies of thetrees for the outbound and return paths have the same configuration.However such configurations are not required. FIG. 58 a and b illustratea situation where the same modules have different routings for thetransmissions from and to the MC. This may be seen by comparing FIG. 58a, which is a return path configuration and FIG. 58 b, where theoutbound paths are shown. The overall round-trip times are illustratedin FIG. 58 c, where the appropriate lengths of dashed and solid linesare used to represent the directions as in the previous examples. In theconfiguration shown, some paths are shown in more than one place, andthe placement of these delay representations is indicated by ellipsesenclosing the line and a line indicating the source of the delay. FIG.59 is the equivalent of FIG. 49 which showed only return paths, but FIG.59 includes a specific tree of outbound paths.

FIG. 60 a, b illustrates multiple signaling paths, where the signalingpaths may be disjoint from the data paths. In FIG. 60 a the module M4may be reached by a path from M0 to M2 to M5 and then M4. This path isshown as terminated with an arrow to distinguish it from the path M0 toM1 to M4, which has previously been discussed. Similarly, module M7 canbe reached by two different paths form M3. The situation where there aretwo different paths from M3 to M7 may represent, for example, an aspectin which there is channelization of a link and each channel has anassociated signal component. Representing the signaling paths in thismanner may account for the situation where the different channels havedifferent signaling delays as they may, for example, use different laneshaving different delays. The additional path to M4 may representadditional connections which exist for communications purposes. Suchuses will be seen as examples, when modules are connected in a gridmesh. In an aspect where more than one module controller is incommunication with a module where there are differing path lengths, thealternative paths may be used to avoid signaling conflicts, as more thanone route may be used to send the control signals.

FIG. 60 b representationally shows the timing as all being associatedwith the path between a module controller MC and each of the modules Mi,where the signaling path is shown as a dotted line and the return dataas a solid line. The additional delay component of delay from module M4is shown twice, and a elliptical symbol indicates the specific instancesthereof. One of the paths is where the signaling was on a path that ranfrom M0 to M1 to M4 and the other path ran from M1 to M2 to M5 to M4.

In the following discussion of detailed scheduling of data or commandsreturning from modules in response to commands, we assume for simplicitythat the commands are transmitted from the MC to all modules with notime delay, and that the addressed module acts on and responds to thecommands with no time delay. This assists in presenting the notation tobe used. Further, a slotted time model will first be discussed, and thisrestriction later shown to be removable. The timing examples in FIGS.61-71 show how the data arrives at the MC.

The time slots are shown on a time line, such that the right hand siderepresents the time of arrival at the module controller MC, and timerunning to the left represents commands or data which will arrive at theMC at a later time. The notation tCi represents the time at which thefirst bit of packet Ci will arrive at the MC. This discussion pertainsto lane 0, for example, and so it should be appreciated that the sametype of scheduling may be performed for each individual lane. Further,the time line showing the delays dMi is shown linearly, however thedelays originate in a tree and may also incorporate variable delayjitter buffers. Thus, the actual time that a module inserts bits onto alane of the link is the delay time of transmission. The effects ofvariation in the time due to the jitter/thermal buffer are not shown asthis would make the figure excessively complex. The result of thissimplification is that if the timing of the bits was considered at alower level of the configuration, the bits may appear to be shifted withrespect to each other such that they would be out of the correctposition on the time line; however as a result of the correction of thejitter/thermal buffers the timing will be correct when received at theMC.

The overall length of a time slot is shown by the large hexagonal blocks900, and the time period of a bit by the small hexagonal boxes 910. Inthis example, a time-slotted system is used, and a module may beconstrained to wait from the time data is ready to transmit until anappropriate slot time is reached. For example, when a read command issent to M7 at t1, the response cannot begin prior to the slot tC_(i+5)as the slot tC_(i+4) has already begun at the module M7. FIG. 61 billustrates the situation after a full slot time has elapsed. At thistime all of the slots have moved in time by one slot time and slotiC_(i+1) is beginning to arrive at the MC. Module M7 has begun insertingdata such that 5 bits have been inserted in slot iC_(i+5). FIG. 61 cillustrates the situation when iC−(i+2) has begun to arrive at the MC,and it may be seen that all of the bits from M7 have now been insertedto a time slot 920. After further time has elapsed, this time slot willbe received at the MC. This occurs where the bits are insertedsequentially at each clock pulse such that the data is filled into aslot.

In an aspect, a fixed time slot may be assigned to each module as shownin FIG. 62 a-c such that the data or commands returned from a module tothe tree root, such as a memory controller MC, arrive in a predeterminedorder. Here, multiple commands have been transmitted by the MC. Again,for simplicity, the commands and the data are considered to be sent in afixed delay system. In this example, the commands are considered to beacted upon as if they had the delay of the furthest module. In such aninstance, even if the are modules with shorter delay, the return isdelayed to fit into the predetermined slot.

FIG. 62 a illustrates the situation where the slot tC(i+6) had arrivedat M7 five clock cycles earlier, and 5 bits of data have already beeninserted into the slot. Additionally, read commands have been sent to M0and M2 and response slots for these commands are assigned to tC(i+6) andtC_(i+7), respectively. FIG. 63 b shows the situation 8 clock cycleslater, where M7 has filled the assigned slot and, additionally, readcommands were sent to M3 and M6 and response slots tC_(i+9) andtC_(i+10) assigned, respectively. After 8 additional clock cycles, theslots have moved by a full slot period. However, it will be noted thatnone of the following slots have reached the device for which they wereassigned to carry a response, as shown by comparing the position of theslots with the delay times dMi, and this no other module has insertedbits.

In FIG. 63 a, 8 additional clocks have occurred, and the slots assignedfor response from modules M0, M2 and M6 progress forward in time, buthave not reached at time at which the respective modules may inert bitsinto the stream on the lane. Eight more clocks have occurred in FIG. 63b, and now the slots for modules M0, M2 and M6 have reached theappropriate module, as shown when comparing with the module delay dMi;at this time module M0 has inserted 1 bit, module M2 has inserted 5 bitsand module M3 has placed 3 bits on the lane. Note that an indication inthe slot arriving at the MC shows the total number of clock periodswhich have elapsed since the beginning of the sequence shown.

At tMC_0=t1+45 module M6 has inserted 7 bits, and the previous slots arealready full. In four more slot times, at the end of slot tC_(i+10), allof the bits will have arrived at the MC.

The total time to perform the sequence illustrated was the time delay tothe furthest module, which was M7 for this lane. For another lane,another module may have the greatest delay. Under such circumstances,the delay may be increased to the greatest delay. In another aspect,FIG. 64 illustrates an example of variable delay scheduling. A slotsystem may still be used, but a command sent to a module may be assignedthe first return time slot that has not yet been assigned and has notreached the delay associated with the module. For comparison, FIGS. 64and 65 show the same time sequences as FIGS. 62 and 63. Here, however,when issuing commands to M0 and M2, the response slots are associatedwith slots tC_(i+4) and tC_(i+5), respectively, rather than slotstC_(i+7) and tC_(i+8). As shown in FIG. 64 b, after 8 clock cycles haveelapsed, commands are issued to modules M3 and M6, and instead ofassigning the response slots as tC_(i+9)and tC_(i+10), the responseslots are assigned to tC_(i+7) and tC_(i+8), respectively. FIG. 64 athrough FIG. 65 c show the time progression of the assigned return slotsand the insertion of data at the appropriate clock intervals. The lastbit of the response to the command sent to module M6 arrives at the MCat tMC_0=t+55, whereas in the previous example using fixed delayscheduling and shown in FIGS. 62 and 63, the last bit of the responseform M6 did not arrive at the MC until tMC_0=t+71. Thus, the delay inresponse from any module may be reduced by using variable delayscheduling.

FIGS. 66 and 67 illustrate a situation where the restriction of data toslotted transmission times has been removed. In FIG. 66 a, a readcommand R was sent to M7 at t=1 (recall that the time delay fortransmission and processing the command has been ignored in thisdiscussion), and the situation is shown 4 clock cycles later, where 4bits of data have already been put on the lane. If a command had beensent to M0 and M2, these modules would be able to insert data as well.FIG. 66 b shows the situation 4 clock cycles later, where all of thebits for the M7 response have been inserted, and 4 bits each for the M0and M2 responses have been inserted. At this time commands are sent toM3 and M6. FIG. 66 c illustrates the situation 4 clock cycles later,when modules M2 and M0 have also completed inserting the response bits,but modules M3 and M6 have not commenced inserting bits. The responseslots for M3 and M6 may not be assigned immediately, as were those forM0 and M2, as they would have overlapped with the response slot for M7,so they may be placed into the next possible clock cycle. FIG. 67 a-cshow the further time progression of the sequence. This situation mayleave some spaces between slots. However, it may be possible to use thespaces for variable width data or commands, or by interleaving responsesfrom different modules.

FIGS. 68 and 69 illustrate a situation for non-slotted responses wherethe signaling delays are shown. The allocation of a time for response isgoverned by the signaling delay where the time for a command to reachthe module is accounted for, and the placement of bits on the lane isdelayed. This may represent a situation where a module may not beassigned a time to put a bit on the lane which will already have passedthe module by the time the command has been received at that module. Itis therefore not necessary for the system to have slotted times.

The module controller MC may be attached to a system or bus running at aslower clock speed than the module controller MC. There may be someperiod of time at which the MC will be receiving high-speed data,de-serializing the data, latching the data and sending it out on anotherinterface, or using the data within the module controller MC. Data whicharrives at the MC with insufficient time to be latched for the lowerspeed clock will become usable at the speed clock cycle of thelower-speed clock.

Transmissions for which the bits do not arrive in time to be used at thenext lower-speed clock cycle may not have any advantage overtransmissions for which the bits arrive at the last possible moment atwhich they may be used in the next lower speed clock cycle, andimprovements to the system performance as well as reductions in logiccomplexity may result.

FIG. 70 a shows a system with the same signaling delays as the system inFIG. 68 a, in which it can be seen that while M3 has a lower signalingdelay (sdM3) than M6 (sdM6), the response to a command sent at time t1would be returned at the same time for both modules, at tL_(t1)+48. As aresult, if the signaling delay of M3 were replaced with an adjustedsignaling delay (asdM3) as shown in FIG. 70 b, the response latency ofM3 remains the same. In this case, the command return scheduler in themodule controller MC may operate without making scheduling decisionsbased on high speed clock times, but on the slower slot times. Thecommand scheduler may also be able to perform scheduling tasks formodules whose signaling delays fall within the same slot time, such asthe case shown where sdM6, asdM5, asdM4 and asdM3 do so in FIG. 70 b.

If such delay adjustment is beneficial to the implementation oroperation of the module controller MC, such adjustment of a modulesignaling delay might be made merely as an “imaginary” adjustment. Thatis, the signal still arrives at module Mi at sdMi rather than at asdMi,but the module controller operates as if the signal was arriving atasdMi.

It is also possible the delay may be increased by performing the typesof lane exchanges shown in FIGS. 35-38 so as to increase the signalingdelay of one module in such a way as to decrease the signaling delay ofanother. The overall signaling delay is the combination of both theforward and backward delays. For example, in FIG. 70 a if by increasingsdM2 by a small amount it were possible to decrease sdM3 so thatsdM3←dM6, then the command latency of M3 would be improved withoutincreasing the command latency of M2.

An improvement in overall delay in one path may arise from adjusting thedelay of more than one other module. Also the improvement in delay mayinvolve increasing the command delay of another module. For example, inFIG. 70 a increasing sdM1 such that sdM1=sdM2, may reduce sdM7 so thatsdM7==sdM6. In this example, the command latency of M1 would increase byone slot time and the command latency of M7 would decrease by one slottime, However sdM1 would still be less than sdM7.I.

An adjustment of delays where the module(s) whose command delays werereduced may actually become less than the module(s) whose delays wereincreased, may also be useful

FIG. 70 c shows an example of the configuration as in FIG. 70 a, wherethe timing at which the module controller MC is capable of transmittingcommands, may not be fixed in relation to the slot times, if any, and/orthe low speed clock. FIG. 70 c shows the situation at t1+4, or 4 highspeed clocks after the example in FIG. 70 a, where the slots have moved4 bits closer to the module controller MC, resulting in a different setof adjusted signaling delay values that may be used without effectingthe command delays of the modules.

FIG. 71 a shows the data delay and signaling delay for modules M0 and M2as in the previous examples, the times the each of the modules wouldtake to execute 3 different commands CMD#1, CMD#2 and CMD#3, as well asthe time taken to execute CMD#3 in the case where the previous commandwas CMD#2, shown as CMD#3_(2). This notation may also account for morethan one prior command or to account for different states of the moduleor system. In this example, the delay for each command is the same forboth modules; however it is not necessary that the delay for a givencommand be identical for each module.

FIG. 71 b shows the resulting command signaling delays, where thenotation “sd[3]M2” is the signaling delay for command CMD#3 for moduleM2 and sd[3_(2)]M2 is the signaling delay for command CMD#3_(2) formodule M2

There may be more than one possible adjusted signaling delay for amodule. For example, the module controller may send the data returnedfrom a module to more than one destination or perform more than one typeof operation with the returned data such that there is more than onepossible clock boundary to be crossed. The clock “boundary” may also bethe start time of a slot in the serialization process leading to adevice, function, process, etc. with a higher speed clock. There is norequirement that the clocks have a fixed phase or frequencyrelationship.

In the preceding examples “high-speed clock” was used, notionally, tomean “a bit time” on the high-speed link, although the high-speed clockmay be different from the bit time if, for example, data is transmittedon both clock edges, multi valued signaling is used, or a differentialclock, a multiphase clock, and/or more than one clock is used. It shouldbe understood that, for purposes of clarity only, the examples have beengiven as if clock times and bit times were the same.

Also, for clarity, to the data has been said to have arrived by “thenext low speed clock”, but this should also be understood to representthe next opportunity at which the data can be used, which may not be theimmediately following clock time.

Examples of physical and logical layouts are given. Other layouts arepossible and will evident to those skilled in the art. For example, FIG.72 shows that a tree, similar to that previously shown in FIG. 5 may belaid out on a mother board or substrate such that modules M0 to M7 aredisposed parallel to each other in a linear fashion. Such an arrangementmay facilitate the flow of cooling air, or optimize the electricallayout of a mother board or main board. In FIG. 72, and following, thelines are represented such that the lines between modules terminate atan arrow, and originate at the boundary of a module to show connectionsbetween modules. Some of these connections pass by intervening modulewithout any functional connection, and such an arrangement may be shownby making the line dotted, or shaded; but, lines originating at a moduleand passing through another module without being terminated by an arrow,are considered to have physically effectively bypassed the module. Thephysical aspects of a line passing by a module without functionalinteraction may depend on the detailed design, but in an aspect thedotted lines or the bypassing lines may represent lines passing on aback side of a printed circuit board, or in a layer of a printed circuitboard or substrate.

FIG. 73 a shows the same arrangement as in FIG. 72, however each moduleis designated as having a “front” side and a “back” side, with thedirection of the front side being shown by a broad arrow. Each modulehas one port on the front side and two ports on the back side, where aport is shown as two unidirectional groups of lines. In another aspecteach port may be comprised of bi-directional lines. An interface betweentwo modules such as M0 and M1 in this arrangement will result in thefront sides facing each other as indicated in FIG. 73 b. Overall, theorientation of modules to form the configuration of FIG. 72 is shown inFIG. 73 a.

FIG. 74 a-d shows examples of other arrangements of modules. In FIG. 746a, modules M0 and M1 are shown in plan view, where the linestherebetween are unidirectional. Alternatively, each of the lines maybidirectional or may be comprised of groups of lines, and the symbols Rand T may be considered as reference designators rather than as receiveor transmit connections. In this instance, the R connection is shown onthe left hand side and the T connection is on the right hand side ofeach module. When connecting the T connection of module M1, for exampleto the R connection of M0, the connection passes beneath each of themodules, whereas the connection of the T connection of M0 to the Rconnection of M1 does not pass underneath the module.

FIG. 74 b shows the arrangement of FIG. 74 a in a profile view wheresome of the lines are routed on one layer of the board 970 a and someare routed on another layer 970 b of the board. Each of the modules maybe a circuit board on which various circuit components are disposed:components may include memory chips, a CSE, other interfaces, computingelements and the like. FIG. 74 c shows another arrangement in plan view,and a corresponding profile view in FIG. 74 d. In FIGS. 74 c, d, themodule is designated as Si, which may represent a switch. A switch S maybe a CSE or other device disposed on a circuit board. However, unlessotherwise specifically restricted, the use of such designators as S, M,T, R are intended as general reference designators rather thanrestricting the function to a switch, a module, a transmit function, ora receive function. Lanes may be unidirectional or bidirectional, and becomprised of groups of lines having a combination of the characteristicsdescribed.

FIG. 75 illustrates an aspect where the interconnections between modulesmay not be on a motherboard or substrate. The connection between M0 andM1 is shown as being on a motherboard, as is the connection between M2and another module (not shown, but disposed to the right of module M2)in the plan view of FIG. 75 a. However the connections between M0 and M2are shown as being by a board 980 connected between the tops of themodules M0, M2, which may be connectorized. Alternatively, thisconnection may be by a cable and connector or other means. Thisarrangement is shown in profile view in FIG. 75 b.

In an aspect, FIG. 76 illustrates a portion of the arrangement of FIG.72, where the connections to the modules are shown as alternating T andR connections on either side of the module for a port. As shown, thereis a single alternation, where the receive connections R are on theright and the transmit connections T on the left, and then transmitconnections T on the left and receive connections R on the right. It maybe seen that three ports are shown on each of the modules. Any number ofsuch alternations of connections may be present, and the number of linesassociated with each alternation may range from zero upwards. The numberof lines in each group may not be equal. Such a disposition ofconnections may eliminate the need to designate a front and a back sideof the module as it may be seen that the modules may be connected asshown, for example in FIG. 73 without rotating the modules end-for-endto orient facing portions. Hence, symmetrical plug-in cards may beproduced for some applications.

In another aspect, FIG. 77 shows the same arrangement of cards, wherethe T and R connections are not alternated. Here the R connection is onthe left-hand side of the board and the T connection is on theright-hand side of the board. This illustrates that alternation ofconnections is not needed for certain arrangements of cards andconnections.

FIG. 78 illustrates a aspect where the board connection configuration ofFIG. 77 is used. The connection from M1 to M3, instead of being carriedon the top two sets of transmission lines comprises the second groupfrom the top. The lane between module M0 and M3 has been routed from thetop of M0 to the switch (not shown) at the center of M0, crosses to thetop of M1 and over to M3 as shown in FIG. 77. Thus, the longest signalpath is half of the length of M0 to get to the switch and then cross toM1, up to the top of module M1. However, by arranging the lanes as shownin FIG. 78, the distance form M3 to the output of M0 the MC (not shown)is reduced. In this aspect only the lengths of the paths on the moduleswas considered. The distance between the modules is also part of theconsiderations of routing. In some situations, the distances traveled onthe modules is greater than the distances between the modules. It may beseen that alternating the paths on the modules reduces the overall delayto modules further down the tree in a manner similar to exchanging theroutes of data logically traversing the lanes.

FIG. 79 illustrates an arrangement of modules having differing widthlinks therebetween. Module M0 has two full lanes with two full ports oftraffic to the module controller (not shown). However M0 has half asmany lanes to each of modules M1, M2, M3 and M4. As each module may fillthe lanes connecting it to M0, two modules may be simultaneouslycommunicated at full speed. In the example of the connections to M5, M6,M7 and M8, having a connection to M1, all of the modules may beaddressed at full speed simultaneously. Such an arrangement may betermed a “fat” tree. In such a tree, the width of the data pathincreases at it progresses to the root so that multiple modules may beaccessed without congestion.

FIG. 80 illustrates a grouping of modules in the form of a grid. In FIG.80 a, four modules are connected in an elementary square, where the fourmodule are mounted so that pairs of modules are parallel to each other,and pairs of modules are in-line with each other. When thisconfiguration is applied to modules being mounted to a board byconnectors, and rising vertically from the board, the alignment of thecards parallel to each other may facilitate the circulation of coolingair. The physical arrangement may thus be made regular, without regardto the logical interconnection arrangement. In the aspect shown, themodules have alternating transmit and receive connections. The U-shapedconfiguration of the lines connecting M0 and M2 may result in a skewreduction as the lane taking the longest route around the outside of theU is the layer which is closer to the center of the module and may becloser to a centrally located switch or CSE.

FIG. 80 b shows a logical representation of the circuits in FIG. 80 a,where each box now represents a module or the like, and a single linerepresents all of the interconnections between modules, such as linkslines, and lanes. From this building block, FIG. 80 c shows how theelemental configurations of 4 modules may be expanded into a grid ofmodules. The grid need not be perfectly regular in any dimension, andmay have gaps in the interior thereof.

A variety of geometrical configurations may be formed, such as the groupof 6 shown in FIG. 81 a. (Note that there is no requirement for themodules to be of dissimilar physical size as shown in the figure; theuse of differing size blocks is for convenience in illustration). Thelogical arrangement of the 6 modules, similarly to the arrangement ofthe 4, is shown in FIG. 81 b and is in the shape of a hexagon. FIG. 82illustrates a grid made up of hexagonal structures. In terms ofconnectivity, as contrasted to FIG. 80 where the path leading from M0goes straight up and connects to the path leading to the bottom of M2,and the transmission lines on the top of the board lead to receive lineson the top of the board on M2, in FIG. 81 a, M0 appears connected to M3and the T line appear to be connected. However M0 is in factcommunicating with M5, and M3 is communicating with M2, and the properterminals are associated due to the 6 module groups being offset fromeach other as shown in FIG. 82 c

In yet another aspect, shown in FIG. 83, an arrangement of modules in agrid configuration and connected to a processor unit (CPU), a memorycontroller or module controller, or other device, which is configured soas to interface to modules. In this instance a memory may be shared with4 central processor units (CPU 1, . . . ,4).

FIG. 84 illustrates yet another configuration where 2 CPU devices areconnected to what may be termed a 2-tree, which may be termed atwo-rooted tree. In an aspect, CPU CO may be operated as the root of atree which is connected to modules M0, M2, M4 and M6, with CPU C1 beingthe root of modules M1, M3, M5 and M7. Access to each of the trees bythe associated root CPU may be performed without regard to the operationof the other tree. However when access to the other tree is desired,then a protocol to arbitrate potential conflict may be provided eitherin the tree systems or between the CPUs. Such dual-port memories maypermit two or more CPUs to communicate by storing messages in the samemodule, providing that an access path to the common module exists. Inparticular, the connection from module M0 to module M1 can bechannelized such that, for example, half of the lines may be under thecontrol of one CPU and half the lines may be under the control of theother CPU such that the CPUs may transmit messages to each other, orotherwise communicate.

FIG. 85 a, b shows two different arrangements of modules connected to aCPU in which a logical tree connects each of the modules to the CPU. Insuch a circumstance, links not on the logical path may not be poweredup; however there is nothing to preclude powering the links. It is alsopossible for different logical trees to be used for control and returnpaths. In such tree arrangements, the data and control flow may beallocated to distribute the load based on usage of individual modules,or the connections may be logically rearranged so as to permit access toI/O devices.

FIG. 86 a shows an array of hexagonal modules, each hexagon having aseparate CPU (CPU 1, . . . ,4), where the control paths are shown as aheavy connecting line. FIG. 86 b shows a grid arrangement of squaremodules with separate CPUs, and for clarity, the logical control pathfrom each CPU is shown with a separate symbology (various solid, dashedand dotted lines). In these figures, the CPUs are shown only accessingan associated group of memory elements, but not sharing access withanother computer. However, FIG. 87 b illustrates the extension of thecontrol and return busses such that they extend into shared memoryareas. The arbitration of memory access to shared memory may beperformed in the trees, or by communication between the CPUs on aseparate communications path, or by some other means. In the case ofCPUs C1 and C2, some of the lanes have been shown as half the widthpreviously shown and, as discussed previously, such a configuration maybe accessed simultaneously by the two processors. The processors mayalso be configured to access the shared memory alternately.

FIG. 88 illustrates a configuration of 4 CPUs and a grid of modules,which may be memory modules, as well as 4 modules configured asinput/output (I/O) devices (I/O#0 through I/O#4). In this instance, itmay be seen that each of the CPUs may simultaneously access any of theI/O modules without the need for arbitration.

FIG. 89 illustrates a configuration were the I/O modules are replaced byother computing system elements such as bus bridges (BB) for connectingto a multi-drop bus, which may be a PCI bus, a point-to-point bus suchas a PCI Express, a memory hub MH #0 interfacing to a multi-drop memorybus and a memory hub MH #1 connecting to a point-to-point memory bus.The width of the lines are shown of unequal width so as to schematicallyindicate that not all paths need be of equal width, nor need the pathsbe symmetrical.

FIG. 90 shows an arrangement of modules and links capable of supportingboth a linear and/or a tree topology, depending on which sets of tracesare used.

FIG. 91 shows the configuration of FIG. 90 used in a linear topologyusing the center traces.

FIG. 92 shows the configuration of FIG. 90 used in a tree topology usingthe outer traces as well as some of the center traces.

FIG. 93 a shows a CSE disposed on a module M with a number of RAM chips,which may be located on both sides of the module, to provide aconnection from the module controller to the RAM chips. RAM is used asan example of a memory type, and should not be interpreted to restrictthe types of memory technologies which may be used.

FIG. 93 b shows 3 CSE's providing the functionality of FIG. 93 a in a‘bit slice’ manner.

FIG. 94 a shows a CSE on a module with a number of RAM chips as in FIG.93 a with the data lines to the RAMs shown as a solid line and theaddress/command lines shown as a dashed line. The size and location ofmodules on the board may not be critical in many applications.

FIG. 94 b shows several CSEs on a module where the data lines from theRAMs have been bit sliced across a number of smaller CSEs. Theright-most CSE on the module controls the address/command lines to theRAMs, although the bit slice CSEs may also control address/commandlines, and the right-most CSE may also carry data from some RAM chips aswell.

FIG. 95 shows a configuration where the relative physical position ofthe signal links on the modules of the leaf modules (M6 and M7) at thebottom of the tree is the same, which may permit various optimizationsto be performed such as building leaf modules with CSEs that may onlyhave pins for one set of links and/or connectors which may only havetraces to connect to one connect one set of links as well as otherbenefits that may be derived from having a module with reducedcapability.

The left side of FIG. 96 repeats the left half of the group of modulesshown in FIG. 95, and illustrates the offset of module M6 from the othermodules with respect to a linear alignment of parallel modules. Theright side of FIG. 96 shows the same modules where the arrangement ofthe traces has been altered so as to reduce the size of the offset ofmodule M6.

FIG. 97 shows an arrangement of modules using bi-directional links.

The term module is used to encompass a CSE and its related functionsand/or components even though the ‘module’ may reside on the same boardas another ‘module’ rather than having to be connected to the modulecontroller and/or other modules through a plugable connector. Theconnections between CSE have been shown as board traces, but this shouldnot be taken to preclude interconnect via wires, cables, opticalinterconnect, substrate trace patterns, and the like”

FIG. 98 shows modules as previously described, with the addition ofcache memory, which may be RAM or other memory technology. The cachesmay be a part of the CSE or the overall module and serve to cache orbuffer data passing through the module on the way to or from the modulecontroller or any other module. The use of cache may facilitateflexibility in the timing of data flow, and if suitable data is stored,the reconfiguration of the data flows to deal with, for example, modulefailures.

FIG. 99 shows a CSE bridging between multiple, possibly differinginterconnect systems. FIG. 99 a shows the CSE connecting to aninterconnect that would be connected to the top edge of the module. FIG.99 b shows a CSE connecting to 2 interconnects that would connect to thebottom edge of the module.

In another aspect, the reliability of systems of communications,computation and memory may be of concern. To the extent that data orfunctionality will be lost due to a failure of one or more components,both the reliability of the individual devices and components, and thearchitecture of the system may be considered to be relevant. Singlepoint failures, such as a computer processor, a memory controller, anode, a memory module, or the like may be considered undesirable.Configurations providing for the correction of data errors and therestoration of corrupted data using redundancy techniques such ascheck-sum, CRC, error correcting codes, and RAID. The term “RAID” isliterally an acronym for “Redundant Array of Inexpensive Disks”, but nowhas a meaning in the art as any scheme for redundancy involving thestriping of data and check bits over a number of data storage devices,and RAID is used in that sense herein. In particular, a communications,computer or memory architecture as described herein may be configured soas to provide both for the use of data correction techniques andredundant links such that more than one failure may be experiencedbefore the overall system is substantially compromised. Such failurescan be detected and components or devices identified and replaced priorto, for example, data loss. In an aspect, the configuration of devicesmay be such that a device, component or module may be replaced while thepower remains applied and the apparatus is remains functional throughoutthe repair. Such a repair is known in the art as “hot swapping”.

FIG. 100 illustrates a binary tree of modules, where the modules aredisposed in a linear fashion. The communications lanes or link or busbetween modules is schematically shown by lines terminated by arrows oneeach end. Where the bus passes under a module but does not connect, thebus may be shown in gray; however, there are no connections to anymodule by any bus except where an arrow on a bus or link touches amodule. In this example, each port is partitioned into two channels andthe channels are shown as one bus on a first side of the module and twobusses on the second side of the module.

In describing a tree arrangement of modules, it is convenient to talkabout both “levels” in the tree and of individual modules by numericdesignation. At the lower portion of FIG. 100, the logical linksconnecting between modules in a binary tree having 5 levels are shown.The root of the tree is a single module at level 0, designated as module0 (M0). The root module may connect to a memory controller MC (notshown) through one port and to two other modules at level 1, M1 and M2,through the other two ports. A module intended for use in a binary treemay have three or more ports and, as discussed previously, the ports maybe divided into one or more channels. In this instance, the ports areshown divided into two channels. It should also be noted that where thehardware and software is configured to support communications betweenany two channels on a module, without regard to port assignment, themotherboard signal line connections may be made to different locationson each of the module connectors. In this example, two channels are usedto communicate between modules directly connecting to each other.

Underneath each of the modules in the linear arrangement at the top ofFIG. 100, a first series of numbers M# indicates the numericaldesignation of module, and the second series of numbers L indicates thelevel of the module in the tree. In this configuration, the root module0 at level 0 is disposed at the center of the group of modules and thebus exiting at the top of the figure connects, for example, to a memorycontroller MC or a CPU or similar device. On either side of module 0 aremodules 10 and 12, at level three. The modules at level 1 are disposedapproximately equidistant between the root module and the two ends ofthe linear physical arrangement of modules. In the situation shown,where modules of equal rank are not disposed opposite each other, thecooling of the memory array may be improved as hot spots may beminimized.

In the tree configuration of the present example, all of the dataeventually flows through module 0 as it is the only module directlyconnected to the MC. This means that the lanes will be energized moreoften than any other module, and the module 0 will have the highestpower dissipation. For convenience in discussion of power dissipation,it is assumed that all of the modules are present, and that reading andwriting is evenly allocated to each of the modules. Modules at level 1therefore may have to transmit and receive half of the data handled bythe root module. The means that the lanes will be energized half asoften, on average, as the root module, resulting in lower powerdissipation in each of the modules; in a simplistic way, half of thepower dissipation of the root module. Similarly, level 2 modules willeach dissipate one quarter of the power of the root module, and level 3modules will each dissipate one eighth of power. Thus, except for theroot module, which dissipates the most power, the other modules eachdissipate considerably less power, and in the arrangement of FIG. 100,the root module is faced on either side by modules at level 3, whicheach dissipate only one eighth of the power of the root module. Specialarrangements may be made for cooling the root module, and the othermodules will be less difficult to cool with few or no hot spotsdeveloping. Later, a means of distributing the throughput load using amulti-rooted tree will be discussed, and it should be evident to aperson skilled in the art that the power distribution of the each rootwill be reduced by a factor of two.

FIG. 101 shows that there are a number of alternative connectionstrategies that may be pursued in a binary tree where there are threeports and each of the ports is divided into two channels. In FIG. 101 a,the channel pairs are configured so as to be disposed symmetrically withrespect to a center line drawn along the length of the lineararrangement of modules. Alternatively, as shown in FIG. 101 b, the samenumber of channels and ports may be configured so that the two channelsconnecting two modules are spaced apart so that at least one channelgoing to another module is interleaved.

A tree may be multi-rooted, and thus have more than one module which maybe considered at level 0. In this circumstance, the individual moduleshave different module numbers in each tree. FIG. 102 shows a first root,where the module numbers M# are listed at the top of the figure, and asecond root, where the module numbers M#* are listed at the bottom ofthe figure. As an example, a module may be number 0 or 11, depending onwhether the module is acting as a root, or as a node in a tree.

Connection of lanes or busses between modules is often constrained bypin densities, by trace, line or lane densities, by cross-talk and thelike. Greater separation between transmit and receive functions andbetween lines is helpful in mitigating these problems. FIG. 103 aillustrates a plurality of modules, disposed symmetrically with respectto a central module, in the plan view that has been used previously,showing an example of the connections using lanes. A possible routing ofthe busses on the printed circuit board PCB is shown in the elevationview FIG. 103 b, where the board has three layers and four surfaces. Thebusses may be conveniently routed so that line crossings, multiple vias,blind vias and the like may be minimized or avoided.

In connectorized motherboard designs, the pins on the connectorspenetrate the PCB and serve to connect with the traces on each layer, sothat the vertical portion of each of the bus is actually a connectorpin. Pins of a connector are usually all of the same physical length andextend through the multiple layers of the circuit board, even wherethere are no connections required on some of the layers. Such pins mayact as radiating points for spurious signals at the high frequenciesassociate with the clock and signal data rates.

While multiple layer PCBs may be useful, there is nothing to precludethe routing of all or substantially all of the bus interconnections on asingle layer. Moreover, the arrangements described herein may bepartitioned in many different configurations. For example, some or allof the components and interconnections, including the CSE and memory,may be incorporated into a semiconductor circuit on a substrate, diesmay be mounted on an interconnection substrate or substrates, and thelike. That is, the construction and packaging of the product may differsubstantially from the specific examples provided herein.

A multiple root arrangement such as is shown in FIG. 104 may be used toachieve some redundancy in the case of a root module failure byprovision of some additional connection traces, and the software andhardware capability to reconfigure the routing in each module. The basictree arrangement is shown as being connected by the solid linesrepresenting busses in FIG. 104 a, and the logical connections are shownin FIG. 104 b. Each block representing a memory module in the tree isdisposed beneath the corresponding memory module in FIG. 104 a. In thissituation, module M0 is the root and is connected to the memorycontroller.

Additional interconnections are provided as shown by the heavy dashedlines in FIG. 104 a, and may be used in the case of a failure modality.In the situation where the port connecting the root module M0 to the MCfails, the tree may be reconfigured as shown in FIG. 104 c. In thiscircumstance, certain of the lanes previously used may not not used(such as the lanes between M0 and M2) and certain of the previouslydormant lanes may be placed in service (such as the lanes between M4 andM5, shown as the heavy dashed line). Concomitantly, the level of theindividual modules in the tree may change. For example, module M0 was atlevel 0, but now is at level 3, and module M4, which was at level 1 isnow at level 0. While providing some redundancy, the arrangement of FIG.104 does not provide for redundancy in the case of failure of any onenode or link.

FIG. 105 illustrates a tree and connections that can accommodate asingle failure anywhere in the tree, without loss of data from otherthan, at most, the failed element. The connectivity to all of themodules and to all of the operable links is maintained. FIG. 105 billustrates the logical arrangement of the modules in a tree, with thesolid link lines representing the baseline condition of the tree withall lanes and nodes operable. The connections shown as dashed lines arethose links which may be dormant, but would be activated in the event ofthe failure of the root module 0. In this circumstance, the modulepreviously designated as 10, at level 3 in the tree is re-designated asthe operating root. This is shown in FIG. 105 b by the dashed lineextending vertically downward from the representation of the module.

FIG. 105 a is a plan view of the mother board lanes and the modules thatis associated with the logical tree of FIG. 105 b. Links shown as solidlines are active when the tree is fully operational with no failures.The links shown as dashed lines are provided so that the tree may bereconfigured to continue to operate in the event of a failure. Theparticular failure shown was for the root module, which may be the mostsevere failure, and the one that may be statistically more likely tohappen as the root module is the module dissipating the most power. Allother factors being equal, the failure rate of a specific type ofsemiconductor circuit increases with temperature. When the tree isreconfigured, not all of the previously operating links will continue tobe used.

The reconfiguration of the tree results in a change in the data andcontrol paths and the level of some of the modules in the tree. It maybe helpful to visualize this transformation by reference to FIG. 106.FIG. 106 a is the baseline configuration where there is no failure, andthe auxiliary links are shown not operating (light dashed lines). When afailure occurs in module 0, consider FIG. 106 b, where the tree has beenflipped so that module 0 is on the bottom and module 15 is on the top.In FIG. 106 c, the operating auxiliary links are shown as heavy dashedlines, and the non-operating links are shown as light lines.

FIG. 107 is a schematic representation of a larger capacity motherboard, where 4 groups of 16 modules are mounted. This is representativean approach to further increasing the capacity of a mother board whilemaintaining redundant operation. Each of the groups of modules, whichmay be designated as a “set” may have the attributes of the module arrayof FIG. 105 and 106. The connection of the four sets is performed bycircuit elements C, which may be CSE devices or similar, so that asingle failure of any of the devices C or the CSE 1 and CSE 2, or of anyof the modules in any of the sets will not result in data loss. Thisattribute also permits hot swapping of modules so that a failed modulemay be replaced without taking the unit out of service.

As previously described, the power dissipation of a module depends onthe level of the tree in which the module is situated, as the level isalso associated with the average data throughput in many circumstances.As the power dissipation at each module location may then be estimated,more effective use of cooling resources may be made and hot spotsreduced. As an example, FIG. 108 shows a plan view of a group of 15modules, similar to previous examples, where the position of the CSE onthe module is shown. The direction of air flow is shown by a broadarrow, and metal fences 1100 are attached to the mother board so as todirect the cooling air onto the modules. The fences extend from themother board to approximately the height of the modules as inserted intothe connectors of the mother board. In the case of module 0 at level 0,which has the highest power consumption, the fence 1100 is disposed sothat a larger opening area is presented to the input air flow and agreater quantity of air will flow over module 0 as compared to module 11and 10, which are both at level 3. Similar allocations may be made butappropriate disposition of the remaining air flow fences 1100, givingmore air flow to modules 1 and 2 at level 1 than to adjacent modules 8,9 12, 13, which are at level 3 in the tree.

Where more than one group of modules is to be cooled, and the coolingair may be routed through a second set of modules prior to beingexhausted from the region of the mother board, a similar approach may beused to even out the temperature distribution of the air. FIG. 109 showstwo groups of 16 modules arranged so that modules at higher levels inthe first tree are disposed behind modules at lower levels, such thatthe sum of the two levels is reduced from a situation where the samemodule configuration is used for both groups of modules. The lower thesum of the module tree level numbers, the higher the power dissipationof the two modules taken as a group. As an example, where module 0 atlevel 0 in the group of modules closest to the air input is in front ofmodule 11 at level 3 of the other group of modules, the sum of the powerconsumptions may be only about 60 percent of that which would obtainwhen the configuration of each group of modules was the same and, forexample, two level 0 modules were in a line with the air flow direction.

In another aspect, as the physical position of the modules on a motherboard is associated with the level of the module in the tree, thespacing between modules may be varied so that the modules at lower ranksin the tree (e.g., the root or roots) are spaced further from adjacentmodules and are expect to dissipate more power than when the modules areat a higher rank in the tree (e.g., leaves). Where more than one modulemay be a root, either in an operating or redundancy situation, the powerdissipation in each configuration may be considered in planning theboard layout.

FIG. 110 shows how different placements and orientations of the CSEand/or heat sinks mounted on the CSE and/or module may provide forimprovements in the airflow and/or the module spacing.

FIG. 111 shows a dual-rooted tree configuration where the nodes havelinks that may be constructed from 12 lanes. The root node has 4 lanesto the MC (not shown), 3 lanes to each of its children and 2 lanes tothe other root. The children of the root have 3 lanes to the root and toeach of their children as well as 3 lanes to the node which is thecorresponding child of the other root.

The physical layout of large memory arrays presents numerous problems,including requiring a large number PCB layers, vias and bends in theconductors constituting the traces on the board layers, or equivalentconsiderations on a substrate. The electromagnetic radiation from thelength of the connector pins, and the effects on ground and power planeintegrity are known to persons of skill in the art. FIG. 112 aillustrates a mask for a single layer of a PCB connecting to fourconnectors and to a board interface connector or an existing memory. Thedensity of traces, and the variation of distance between the traces,which may be associated with signal coupling and propagation timevariations, and the deviation from straight line traces, resulting inimpedance variations for high frequency signals, and which may result insignal waveform distortion are also known. FIG. 112 b is an example ofthe details of the layout of the layer in FIG. 112 b. The circlesrepresent the pins of the connector, where the dark circles areassociated with connections to traces on the board layer shown, and thelight circles are pins passing through the layer and connecting totraces oh another layer. Far from the pins, the traces are disposed withvarying distances between adjacent traces, and some of the traces areclose together, increasing the coupling. In the vicinity of the pins,groups of traces are routed between pins, reducing the distance betweentraces and between the traces and the pin. In this region as well thereare multiple bends in many of the traces, resulting in impedancediscontinuities. FIG. 112 c illustrates the paths which may be taken byconnections between two connectors A and D, where the traces must passbetween the pins of intervening connectors B and C. The pairs of tracesshown may represent the layout where differential signaling is used. Asecond layer is also shown, where the signals may travel along pathsthat bring portions of the signal traces on one layer close to those ofan adjacent layer. This may pose problems in board layout.

FIG. 113 illustrates a method of mitigating board layout constraints. Inthis example, each connector 1200 is rotated on the board so that theconnector is oriented at an angle of approximately 30 degrees withrespect to the parallel arrangement of connectors in many memory boards,or similar applications. When the connectors are arranged in a cantedmanner as shown, the traces 1300 can be routed between non-contiguousconnectors with little or no bending to avoid the connector pins 1400.The angle of canting depends on the spacing and number of rows. Morethan one minimal bending path exists in this arrangement and FIG. 114illustrates a configuration where two paths with minimal bending arepossible, for a group of 240-pin connectors. A slight change in spacingbetween adjacent boards or in the spacing of the pins may furtherminimize the required bending.

FIG. 115 illustrates an arrangement of connectors on a motherboard toaccommodate 84 DIMM boards disposed in a canted manner. Only minimalboard area may be left unoccupied when a large number of connectors aredisposed thereon, and such unoccupied area may be conveniently used forboard interfaces, memory controllers, ancillary services, filtering andthe like. Such canting of the connectors may be used with any of theconnector and module arrangements described herein. For simplicity ofpresentation, the connectors in the other examples are oriented in aparallel manner.

Large memory boards may be laid out using the apparatus and methodsdescribed herein, and FIG. 116 represents an arrangement of 82 DIMMmodules configured as 64 DIMM memory modules with RAID 5 errorcorrection capability in a 4+1 configuration, with 2 spare DIMM modules.The memory board supports two trees, connected at the leaves. Even withthe loss of any two modules, all of the other modules may be accessed byappropriate reconfiguration of the remaining data and control paths.Thus, maintenance of the memory board may be performed before data isunrecoverably lost, and may be performed by hot swapping of the failedmodule for a new module. Where the hot swapping is performed when onlyone DIMM has failed, the configuration prevents data loss in the eventthat the maintenance technician removes the wrong module forreplacement. In this example, the logical arrangement is shown in FIG.116 and the corresponding board layout in FIG. 117.

In FIG. 117, the solid lines represent the baseline tree configurationand the dashed lines are links that need not be used in the baselineconfiguration, but may be used when a module failure occurs and are-configuration of the tree is needed to maintain connectivity betweenthe roots and the remaining modules. The dashed lines represent linksthat are capable of operating at half of the throughput of the solidlines. At level 0 of the tree shown with solid line connections, eachtree has a root and this is shown as connecting to the memory controller(not shown). In another aspect, the arrangement may be considered asfour sub trees, with pairs of sub-trees being combined to form eachtree. Each of the sub-trees in the upper tree is a mirror symmetricversion of the other tree, where the line of symmetry passes through thenode acting as the root node to combine the sub-trees. The lower treehas similar mirror symmetry with respect to the associated root node. Inaddition to the ancillary connections in each tree providing forredundancy, the leaves of the two trees are interconnected withancillary connections, including the designation of a module as a rootof the redundant tree, that module having a connection to a MC, whichmay be a redundant MC.

FIG. 118 shows another 82 module tree having two roots. This tree willbe utilized to illustrate an example of a mother board layout techniquesuitable for large interconnection systems. Such techniques may be usedon interconnection substrates and within semiconductor circuits or diesas well. The term “motherboard” is used for convenience only.

Each of the two trees has a depth of 5, and a first configuration usesroots R1 and R2 to connect to a memory controller MC. A redundantconfiguration may be formed using one of R1′ and R2′ in place of afailed root to continue to operate with no data loss. The links shown assolid lines represent the first configuration, and the links shown asdashed lines are provided so that alternate configurations may be usedin the event of the failure of two or fewer modules. The mother boardlayout may be considered as two similar interconnected trees. An uppertree, which may be considered to be the modules connected to R1 by solidlines, and a lower tree which may be considered to be the modulesconnected to R2 by solid lines.

FIG. 119 shows schematic representation of a physical layout of theupper tree connectors and the corresponding portion of the tree logicaldiagram. The module R1 is at level 0 of the tree, and in this example islocated at the center of the row of connectors, and has one link to thememory controller MC. The MC is not shown and may be located whereverconvenient, either on or off of the mother board. The links between themodules of this binary tree are coded by shading and there are threelevels of shading. For convenience they are associated with colors,being blue (the darkest), green (intermediate) and red (lightest). Eachof the shaded broad lines represents a link between two modules that areconnected to the ends of the lines. Lines not terminating at a modulepass beneath the interposed modules without connecting thereto. Thelinks have the properties of links previously described and the broadline may represent bidirectional lanes, groups of unidirectional lanes,channelized lanes, and the like for making connections between modulesand connectors.

Most of the links may be represented by straight lines, and very few ornone of the links need cross each other. This simplifies board layout,enables reduced signal coupling and may reduce the number of PCB layersneeded to construct the board. Links such as those designated a and b,which connect the root module to the first level in the tree are shownas being green. After the cross-sectional aspects of the board arediscussed, a person of ordinary skill in the art will appreciate thatthe links a and b may be routed on the red level of the PCD, with ashorter length and fewer trace bends. Hence, the board layouts shown maybe susceptible to further optimization, and are merely illustrative ofthe ability to simplify the arrangement and interconnection of largenumbers of connectorized memories, or of memories mounted to a circuitboard or incorporated in an integrated circuit, on a substrate, or in apackage.

Cross-sections of the mother board, encompassing the modules locatedbetween I-I are shown in FIG. 120. There are three relevantcross-sections A-C, representing three groups of traces, being linksconnecting to the modules. It should be understood that the broad linesare schematic representations of the lines in the links and may connectto pins of the connectors that may be uniformly or non-uniformlydistributed on the circuit card. Further, the links of different colorsmay lie directly above each other, and are separated in the drawing inorder to facilitate description. FIG. 120 a shows the plan view of themotherboard so as to identify the cross-sections being described. Forconvenience the individual circuit cards are assigned number of 0-7, andthe corresponding numbers are used for all of FIG. 120. In FIG. 120 b,cross-section A of FIG. 120 a is shown; cross section B is shown in FIG.120 c; and, cross section C is shown in FIG. 120 d. In cross-section A,blue, green and red links are shown. The printed circuit board is shownas having two ground plane layers and a VCC (power) layer and the tracesare disposed, using PCB technology in this example, so that a portion ofthe traces (lines) for each link are disposed on opposite sides of theground or VCC layers. This may result in increased isolation betweentransmitting and receiving traces. The horizontal shaded lines thusrepresent a possible disposition of the links between the connectorsshown where the links do not cross each other, and may be laid out in agenerally straight direction. As shown, each of the links terminates ina pin, representing a group of pins, associated with the connector forthe module to be connected, and the vertical shaded lines are associatedwith the connector pins.

In FIGS. 120 c and d, the routing for cross-sections B and C are shownto have similar properties.

The example used connectors with pins, but surface mount technology(SMT) connectors may also be used. For SMT connectors, the horizontaltraces rise to the connector using through-plated vias in place of thepins. However each of the vias may not have to extend the full thicknessof the board to reach the desired trace, and the vias may beback-drilled to remove the plating in board layers more distal from theconnector than the trace to be connected. This minimizes the length ofthe vertical conductive elements, which may reduce radiation andcoupling between traces while avoiding “blind” vias. In addition, thefewer the number of layers involved in interconnection, a tolerance maybe used in the alignment thereof, and this may be used to minimize therelief provided in the ground and Vcc planes, thus providing increasedshielding. While this example showed the traces for the links to bedisposed in three colors, it is equally possible to lay out the boardwith only two colors, although some bending of the traces may result.

Many alternative mother board layouts are possible, and two additionallayouts are shown as examples. In FIG. 121 another 82 DIMM logicalarrangement is shown where the tree is intended to be laid out in threerows of connectors. This tree may also be seen to have furtherredundancy characteristics.

FIG. 122 shows the layout corresponding to the logical tree of FIG. 121,using the same notation as in FIG. 119. The spacing between adjacentconnectors may be adjusted. For, example, the spacing between modulesdesignated as root modules or redundant root modules may be increased toaccommodate the additional power distribution of a module when acting asa root. Also, the modules in the center row may be grouped more closelyto provide space for, for example, module controllers, CPUs and thelike. In an aspect, a root module may be replaced by a CSE withoutstorage memory, or with minimal memory to reduce the power consumptionof the module when acting as a root.

When an 82 DIMM motherboard is fully populated with DIMMs, andconfigured as 62 memory modules with RAID 5, and each DIMM used has, forexample, 2 Gbytes of memory, the capacity of the motherboard will be 164Gbytes. Such a motherboard may have dimensions of approximately15.4×21.4 inches. Of course other DIMM memory capacities and othermemory types may be used in a similar manner. The example given ismerely for illustrative purposes. Various redundancy methods as areknown in the art may be used on each of the DIMM modules, a plurality ofDIMM modules may also be used in achieving redundant operation wheninterconnected and operated as described herein.

In FIG. 123, another 82 DIMM logical arrangement is shown where the treeis intended to be laid out in three rows of connectors, and to make useof the front-to-back cooling arrangement as described in conjunctionwith FIG. 109. FIG. 124 shows the corresponding motherboard layout.

The motherboard link layouts are seen to be formed from a small numberof unit cells, each associated with a module as shown in FIG. 125. Cellsdesignated A through H represent the module types that may be used in abinary tree layout. The cells are schematically represented by avertical line representing the connector and three horizontal stubs,each stub representing a link or any of the types previously described.The lines in the link may be grouped together, separated into channels,or individually routed, and the cell types are merely indicative of thedirection that the links take with respect to the connector. Moreover,the cell designation does not limit the side of the connector throughwhich the traces connect to the memory module. An example of a 16connector layout is shown and most, but not all of the cell types areused in the layout. The cell type associated with specific connectors isindicated. Where less than three links are used to connect the connectorto the tree, a cell type is not shown, as several of the cell types,with one link removed, would permit the arrangement shown.

In memories disposed as trees, clocks may be local, area or systemclocks, RAMBUS clocks or the like, and combinations of clockingtechniques. For example, each of the DIMM modules may have a local clockfor managing the memory and performing local processing functions. Ameso-synchronous clock may be provided by distributing a central clockto a plurality of the memory modules, where the phase of the clock atthe individual memory module CSE is not known, but the phase is assumedto be slowly varying. In such a circumstance, only the phase need beresynchronized when a link is activated, and the startup time may bemade short. The phase may need to be recovered on each of the lines ofthe link, but a group of lines may exhibit similar phase variations andmay be adjusted as a group. In another aspect, the clock may bedistributed on the address lines, one of which is active between alllinks at all of the times. Alternatively, one of the links may be usedto transmit signals such as NOP commands when the interval between data,command or address data has not been transmitted to or from the adjacentmodules for a predetermined period of time.

In another aspect, a memory controller MC may be used to interface toone or more root modules on one hand, and directly or indirectly toanother communications bus or a computer element on the other. Theinterface between the MC and the root module may be by way of a link ofthe type the root module uses to communicate with other modules in thetree. Alternative a one or more CSEs may be provided between the MC andthe modules of the tree, where one of CSEs performs the function of theroot module or a node so that that a module with lower power consumptionmay be obtained at a point in the tree where the CSE has the highestthroughput, A MC may perform the functions of, for example, de-skewing,addressing, frame management, buffering of data and commands in theupstream and downstream directions, initialization, including skewdetermination, module integrity and other turn-on tests, andreconfiguring the trees in the event of a module failure. The MC mayalso manage the power consumption during turn on, so that the peak powerrequired by the attached memory is consistent with the capability of theassociated power supplies.

Although the present invention has been explained by way of the examplesdescribed above, it should be understood to the ordinary skilled personin the art that the invention is not limited to the examples, but ratherthat various changes or modifications thereof are possible withoutdeparting from the spirit of the invention.

1. An interconnect system, comprising: a plurality of nodes, each nodehaving a port having at least one of an input link or an output link,each link further comprising a first lane and a second lane; a pluralityof connectorized memory modules; and a motherboard having a plurality ofconnectors, each of the plurality of connectors interfacing with one ofthe plurality of memory modules, wherein one of the plurality of memorymodules communicates with at least one of the plurality of nodes;wherein each lane further comprises at least one of a signal receiver ora signal transmitter; and wherein a first lane of a first node transmitsa first signal, and a second node is configured such that the firstsignal received on a first lane of the second node controls a futuretime power status of a second lane of the second node.
 2. The systemaccording to claim 1, wherein a node of the plurality of nodes ismounted to the memory module.
 3. The system according to claim 1,wherein the second node has a first port and a second port and the firstlane of the second node and the second lane of the second node areassociated with differing ports.
 4. The system according to claim 1,wherein the future time has a value that is one of a time dependent onthe position of the node in the interconnect system, or a time valuecontained in the control signal.
 5. The system according to claim 1,wherein the future time value associated with each of a plurality ofsecond lanes is separately specified.
 6. The system according to claim1, wherein the future time value associated with each of the pluralityof nodes is configurable.
 7. The system according to claim 1, whereinthe first signal received by a first port controls the status of asecond port.
 8. The system according to claim 7, wherein the status ofan output lane of the first port is changed at a future time determinedby the first signal.
 9. The system according to claim 1, wherein eachnode is configured such that the data second signal controls adestination of data received by the node.
 10. The system according toclaim 9, wherein the destination is selected from one oflocal-to-the-node, or a port.
 11. The system according to claim 1,wherein the first signal received by a first port controls the status ofan output lane of the first port.
 12. An interconnect system,comprising: a plurality of nodes, each node having a port having atleast one of an input link or an output link, each link furthercomprising a plurality of lanes; a plurality of connectorized memorymodules; and a motherboard having plurality of connectors forinterfacing with the plurality of memory modules, wherein the node isconfigurable such that a first signal is received on a selected lane ofthe plurality of lanes.
 13. The interconnect system according to claim12, wherein a node of the plurality of nodes is mounted to a memorymodule of the plurality of memory modules.
 14. The interconnect systemof claim 12, wherein a second signal is received on another lane of theplurality of lanes.
 15. The interconnect system of claim 13, wherein thefirst signal and the second signal are received on separate lanes of theplurality of lanes.
 16. The interconnect system of claim 12, wherein thenode configuration is reconfigurable.
 17. The interconnect system ofclaim 12, wherein the first signal and the second signal are digitaldata.
 18. An interconnect system, comprising: a plurality of nodes, eachnode having a port having at least one of an input link or an outputlink, each link further comprising a first lane and a second lane; amotherboard having a plurality of connectors for interfacing with aplurality of memory modules; and controlling the future time status ofthe function of a second lane based on a signal received by the firstlane.
 19. The interconnect system according to claim 19, wherein thestatus change occurs between states selected from one of powered on,powered off or standby.
 20. The interconnect system according to claim20, wherein the status change occurs between states selected from one ofpowered on or powered off.
 21. The interconnect system according toclaim 20, wherein the powered on state permits at least one oftransmission or reception of the first or the second signal, the standbystate has lower power consumption than the powered on state, and thepowered off state has lower power consumption condition, with respect tothe standby state.
 22. An interconnect system, comprising: a pluralityof signal lanes, each lane capable of at least one of transmitting orreceiving a data signal comprising a bit of a data word, wherein thedata word has a first bit and a second bit and each of the first bit andthe second bit are logically exchanged between two signal lines suchthat a differential time delay between the first bit and the second bitis altered when the word is received at a distal end of the signallanes.
 23. The interconnect system of claim 23, wherein the logicalexchange is performed at a proximal end of the signal lines.
 24. Theinterconnect system of claim 23, wherein the assignment of the logicalexchange is performed at the distal end of the signal lines.
 25. Theinterconnect system of claim 23, where the logical assignment of bits atthe distal end is altered such that the bits are in the same logicalorder as at the proximal end.
 26. An interconnect system, comprising: aplurality of nodes, the nodes forming a network connected by linkshaving lanes; and a motherboard having a plurality of connectors forinterfacing with a plurality of memory modules, wherein a first node isa source of data, and a second node is a destination of data, and atleast one of the first node or the second node is configured to exchangethe logical assignment of data to lanes such that the difference inarrival time of data on the lanes at the second node is changed.
 27. Theinterconnect system according to claim 27, wherein a node of theplurality of nodes is mounted to a memory module of the plurality ofmemory modules.
 28. The interconnect system of claim 23, wherein thedata comprises a word having at least two bits, and a first bit isassigned to a first lane and a second bit is assigned to a second lane.29. The interconnect system of claim 23, wherein the difference inarrival time is altered at a location of the second node where the datais read.
 30. The interconnect system of claim 23, wherein the differencein arrival time is altered at a location of the second node where thedata is deskewed.
 31. The interconnect system of claim 27, furthercomprising a module controller, wherein the logical assignment of bitsof a word to lanes at the input to the first node is such that the bitsare received in a predetermined logical order at the second node. 32.The interconnect system of claim 23, wherein at least one of the modulecontroller, the first node or the second node is configured such thatthe received data at the second node has the same logical ordering as atthe input to one of the module controller or the first node.
 33. Theinterconnect system of claim 23, wherein the logical assignment isperformed at a sending end of the link.
 34. The interconnect system ofclaim 23, wherein the logical assignment is performed at a receiving endof the link.
 35. The interconnect system of claim 23 wherein the logicalassignment is performed at a third node disposed between the first nodeand the second node.
 36. A node, comprising: an input port, and anoutput port, each of the input port and the output port having aplurality of lines; and a switch configured such that each of pluralityof input lines is connectable to one of a plurality of output lines,such that a time skew between signals on the plurality of output linesis altered when measured at a second node having an input port havinginput lines communicating with the distal end of the output lines fromthe output port of the first node.
 37. An interconnect system,comprising: a plurality of nodes, the nodes forming a network connectedby links having lanes; a first node being a source of signals, and asecond node being a destination of signals; and means for reducing adifferential time delay between signals on two or more lanes between thefirst node and the second node.
 38. A memory module, comprising: aread-write memory device; and a configurable switching element (CSE),further comprising: a first port for at least one of receiving ortransmitting signals, the first port having a plurality of signal lanes;wherein the CSE is configurable to interpret received signals to performat least one of: change the state of a signal lane of the plurality ofsignal lanes at a future time, and route received signals to one of theread-write memory device or a second port.
 39. The memory moduleaccording to claim 39, wherein the future time has a value that is oneof a time value determined by the control signal or a configurablevalue.
 40. The memory module according to claim 40, wherein the futuretime value associated with each of the plurality of lanes is separatelyspecified.
 41. The memory module according to claim 39, wherein thefuture time value associated with each of the plurality of nodes isconfigurable.
 42. The memory module of claim 39, wherein the state ofeach of the plurality of signal lines is one of powered up, or powereddown.
 43. The memory module of claim 39, wherein the state of each ofthe plurality of signal lines is one of powered up, powered down, orstandby.
 44. The memory module of claim 39, wherein the CSE isconfigurable such that a logical data assignment received by thereceiving signal lines and a logical assignment of data to betransmitted by the transmitting signal lines is alterable.
 45. Thememory module of claim 39, further comprising a third port, wherein theCSE is configurable such that a signal received on a signal line of anyof the first, the second or the third ports is transmitted on at leastone of the of signal lanes of the first, the second or the third ports.46. The memory module of claim 39, wherein at least one of the pluralityof the transmitting and receiving signal lines is reconfigurable as areceiving or a transmitting signal line, respectively.
 47. The memorymodule of claim 39, wherein a signal is transmitted on at least one ofthe lanes during a fixed time interval.
 48. The memory module of claim47, wherein the fixed time interval is determined such that at least oneof data edge tracking, clock, clock phase, or frame synchronization ismaintained.
 49. A memory module, comprising: means for storing data;means for controlling the future time status of one or more of an inputor an output interface; and means for routing data to at least one ofthe output interface or the means for storing data.
 50. The memorymodule of claim 50, wherein the means for routing data is a configurableswitching element (CSE).
 51. The memory module of claim 50, furthercomprising: means for exchanging a logical and a physical interfaceassignment for a signal so that a differential time delay betweenlogical signals on an output interface of a first memory module isaltered when measured at an input interface of a second memory module.52. A configurable switching element (CSE), comprising: at least twosignal lanes; a first signal lane configured to receive a first signal;and a second signal lane having a controllable state, and a means ofexchanging data between a first second signal lane and a second secondsignal lane.
 53. The CSE of claim 53, wherein the state of a second laneis one of powered on, powered off, or standby.
 54. The CSE of claim 53,wherein the state of a second lane is one of powered on, or powered off.55. The CSE of claim 53, wherein the first signal received by the firstsignal lane controls the state of the second signal lane.
 56. The CSE ofclaim 53, wherein an second signal is received on the first signallanes, and a second signal received on the second signal lane is routedto at least one of another signal lane or a memory interface.
 57. TheCSE of claim 57, where the memory interface is connectable to a meansfor storing data.
 58. A configurable switching element (CSE),comprising: a first port for at least one of receiving or transmittingsignals, the port having a plurality of signal lanes, wherein the CSE isconfigurable to interpret \a received first signal to perform at leastone of: change the state of at least one of the plurality of signallanes based on the received first signal, or route a received secondsignal to an interface.
 59. The CSE of claim 59, wherein the interfaceis a memory interface.
 60. The CSE of claim 59, wherein the CSE isconfigurable to route the received second signal to a second port. 61.The CSE of claim 59, wherein the state of each of the plurality ofsignal lines is one of powered up, powered down, or standby.
 62. The CSEof claim 62, wherein the state of each of the plurality of signal linesis one of powered up, or powered down.
 63. The CSE of claim 59, whereinthe CSE is configurable such that a logical data exchange between thelogical assignment of data on receiving signal lines and the logicalassignment of data of to transmitting signal lines is performed.
 64. TheCSE of claim 59, having a third port, and configurable so that a signalreceived on any signal line of any of the first, the second or the thirdports is transmitted on any of the plurality of signal lines of thefirst, the second or the third ports.
 65. The CSE of claim 54, whereinone or more of the plurality of the transmitting and receiving signallines is reconfigurable as a receiving or a transmitting signal line,respectively.
 66. A configurable switching element (CSE), comprising:means for controlling the future time status of one or more of an inputor an output interface; and means for routing data to the outputinterface or to a means for storing data.
 67. The CSE of claim 67,further comprising: means for exchanging a logical and a physicalinterface data assignment for a signal so that a differential time delaybetween signals on a first output interface and a second outputinterface is altered when measured at a second interface of a secondCSE.
 68. A interconnection network to accommodate nodes in a network ona motherboard, comprising: connection interfaces having a top end and abottom end, comprising a plurality of signal lines forming first, secondand third link connections; and a network formed by one or more of thefirst, second or third links joining the connection interfaces whereinat least two of the following configurations of the connectioninterfaces are joined by traces: (a) the first through third links beingdisposed such that all of the links lead away from the connectioninterface on a first side; (b) the first through third links beingdisposed such that a link having a line closest to the top end leadsaway from the first side of the connection interface, and the tworemaining links lead away from the second side of the connectioninterface and the links leading away from the second side of theconnection interface are disposed adjacently; (c) the first throughthird links being disposed such that a link having a line closest to thebottom end leads away from the second side of the connection interface,and the two remaining links lead away from the first side of theconnection interface and the links leading away from the first side ofthe connection are disposed adjacently; (d) the first through thirdlinks being disposed such that a link having a line closest to the topend and a link having a line closest to the bottom end lead away fromthe first side of the connection interface, and a link having a linedisposed intermediate between the first two links leads away from thesecond side of the connection interface; and (e), (f), (g), and (h),wherein (e), (f), (g) and (h) are (a), (b), (c), and (d), wherein thetop and bottom ends are interchanged by rotating the pattern of (a),(b), (c) and (d) by 180 degrees about an axis orthogonal to theconnector.
 69. The interconnection network of claim 69, comprising atleast three of the configurations (a) through (h).
 70. Theinterconnection network of claim 69, wherein the links are disposed in aflat printed wiring assembly.
 71. The interconnection network of claim69 wherein the links are disposed as conductive metal elements in layersof a substrate.
 72. The interconnection network of claim 69, wherein atleast memory circuits are disposed on the substrate.
 73. Theinterconnection network according to claim 69, wherein at least threeconnector interfaces are disposed parallel to each other.
 74. A methodof laying out a network interconnection, the method comprising:disposing a plurality of connection interfaces such that the tops of theconnection interfaces are positioned opposite each other and the bottomsof the connection interfaces are disposed opposite each other; andforming a network of connections between the plurality of connectioninterfaces, where at least two of the following connection interfaceconfigurations are used: (a) the first through third links beingdisposed such that all of the links lead away from the connectioninterface on a first side; (b) the first through third links beingdisposed such that a link having a line closest to the top end leadsaway from the first side of the connection interface, and the tworemaining links lead away from the second side of the connectioninterface and the links leading away from the second side of theconnection interface are disposed adjacently; (c) the first throughthird links being disposed such that a link having a line closest to thebottom end leads away from the second side of the connection interface,and the two remaining links lead away from the first side of theconnection interface and the links leading away from the first side ofthe connection are disposed adjacently; (d) the first through thirdlinks being disposed such that a link having a line closest to the topend and a link having a line closest to the bottom end lead away fromthe first side of the connection interface, and a link having a linedisposed intermediate between the first two links leads away from thesecond side of the connection interface; and (e), (f), (g), and (h)wherein (e), (f), (g) and (h) are (a), (b), (c), and (d), wherein thetop and bottom ends are interchanged by rotating the pattern of (a),(b), (c) and (d) by 180 degrees about an axis orthogonal to theconnector.
 75. A network controller, comprising: a first interfacecommunicating with a computer; a second interface having to a link forcommunicating with a node; and means for determining the differentialtime delay between signals on lanes of a link disposed between a firstnode and a second node, and means for assigning logical signals to lanescommunicating between first and second nodes to alter the differentialtime delay between the logical signals received at the second node. 76.A network controller, comprising: a computer interface; a first datainterface, capable of transmitting and receiving signals; and aprocessor configured to generate address values for transmitting by thefirst data interface, the processor further configured to logicallyassign data values to signal lines such that a delay differential of atleast two signal lines is altered when received at a receiving device.77. A node, comprising: an input port, and an output port each of theinput port and the output port having a plurality of lines; and a switchconfigured such that each of plurality of input lines are selectivelyconnected to one of a plurality of output lines, such that a skewbetween signals on the plurality of output lines is altered whenmeasured at a second node having an input port communicating with thedistal end of the lines from the output port.
 78. The node of claim 78,further comprising a first and a second output port, the first outputport configured for connecting to a memory.
 79. An interconnect system,comprising: a tree of nodes; each node having a link having a pluralityof lines and provision for attaching memory; a motherboard having aplurality of connectors for interfacing with a plurality of memorymodules; wherein the link further comprises a first line and a secondline, the first line capable of transmitting control data, the nodebeing configured so that control data is capable of controlling thepower status of the second line.
 80. The system according to claim 80,wherein the node is configured such that the control data determines thedestination of data received by the node on the second line.
 81. Aninterconnect system, comprising: a plurality of signal lines, and amotherboard having a plurality of connectors for interfacing with aplurality of memory modules, wherein a word having a first bit and asecond bit associated with at least two signal lines is logicallyassigned each of the two signal lines so that the differential timedelay is altered for bits received at a distal end of the signal lines.